Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 8.607435367s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:44:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 8.607465545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 6.187589252s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:44:10 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.187623959s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.055206729s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:43:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.055231928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.3212138s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:43:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.321242144s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 7.007605365s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:43:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 7.007706038s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.117290496s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:43:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.117514674s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 6.076296378s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:43:30 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.076373184s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 7.047570851s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:43:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 7.047634421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.153065933s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:43:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.15311228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 12.301139919s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:43:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 12.301220402s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 6.393282854s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:42:55 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.393328972s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.228969787s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:42:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.229003251s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 7.426838374s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:42:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 7.42686838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.25294809s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:42:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.252998394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.119678385s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:42:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.119713342s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 6.223321122s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:42:06 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.223434627s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.159469585s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:41:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.159503028s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.334062968s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:41:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.334093427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.106603025s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:41:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.106641038s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.922942768s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:41:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 5.922972727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.X................................] duration: 6.500185409s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:41:27 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.500215758s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.930190691s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:41:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.930288648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.177110791s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:40:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.177191634s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.475810038s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:40:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.475838781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.144442142s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:40:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.144638176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.246665365s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:40:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.246701024s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration: 6.071591709s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:40:26 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.071636043s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.282862419s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:40:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.282897094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.290408471s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:40:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.290461443s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.10800662s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:39:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.108036287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 7.190620948s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:39:49 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 7.190664119s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 7.400448702s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:39:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 7.400475874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 10.543416715s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:39:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.543452754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 6.372524126s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:39:27 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.372559915s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.306675494s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:39:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.30672039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.15798169s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:39:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.158024773s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 6.520680872s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:38:59 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.52071624s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.02889741s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:38:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.028937677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.989100322s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:38:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.989130909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.30177941s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:38:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.301808727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.18920288s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:38:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.18922957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.19262091s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:38:14 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.192655536s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 14.533750143s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:37:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 14.533778867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.863861165s trace: https://jaeger.dev.storj.io/trace/d53ad46a3bad038 FAIL (2025-11-13 14:37:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ... Nov 13 14:17:39.718806610 (102h13m38.270277071s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.718869189 (102h13m38.27033964s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719243782 (102h13m38.270714243s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719600372 (102h13m38.271070833s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719757630 (102h13m38.271228102s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.719982179 (102h13m38.27145265s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720312697 (102h13m38.271783169s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720732096 (102h13m38.272202567s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.720836485 (102h13m38.272306946s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721147126 (102h13m38.272617587s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.721537239 (102h13m38.2730077s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.722351200 (102h13m38.273821661s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723267113 (102h13m38.274737584s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.723886824 (102h13m38.275357285s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724315690 (102h13m38.275786151s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.724792528 (102h13m38.276262989s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:39.725327326 (102h13m38.276797787s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Uploading piece... Nov 13 14:17:41.027262394 (102h13m39.578732845s) [upload=89995]: Next returned nil segment Nov 13 14:17:41.027282332 (102h13m39.578752783s) [upload=89995]: Waiting for error group managing segments... Nov 13 14:17:41.357905237 (102h13m39.909375698s) [noise=true , address=64.49.13.26:4948 , piece_id=MQ6RQQQ36YHFSTJYDE44AATC7LQPIFCB7QOS72DMTWRETZVAQWDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.433145731 (102h13m39.984616192s) [noise=true , address=64.49.13.154:4929 , piece_id=2K55IIOVKBDVMEO3QKXXOZ65X35SQIVSEN6RJAW6OWEGE6ZKYCRA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.460916682 (102h13m40.012387133s) [noise=true , address=5.231.73.2:4018 , piece_id=KP2VQBGZYKUDBJNAIKXV6VRHPK6V3UI6X3SD6HIJOHDTZ6Z6EGLQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.497737346 (102h13m40.049207797s) [noise=true , address=5.231.73.5:4019 , piece_id=H6CZGSTCCYMOZVTSJ2Y5XAKGZA26EQ7E2NONL6DWZGC7Q2LORHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.511205242 (102h13m40.062675703s) [noise=true , address=5.231.73.3:4030 , piece_id=O4B5VTAWE2WNRG2GZT4OYOEGZLDNYDTHZOT3WETYA3P4XHQR5AOQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.536492093 (102h13m40.087962565s) [noise=true , address=5.231.73.4:4022 , piece_id=GABEYHU5QIUMMQYAHKVXLY6QFY4AZRXMVPSDJ3EO6WOIKA7E4U4Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.553999952 (102h13m40.105470433s) [noise=true , address=64.49.13.138:4492 , piece_id=CXNCL6OGRP6C4BXBSH7URFSYJELZLM62KVHGCDJQPD7FVTB67Y3A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.668337736 (102h13m40.219808187s) [noise=true , address=45.91.116.7:4016 , piece_id=BXDTJNCLTSVXQUCC4RO2C2JHZ26XHIJPTHITUO7TXQY6CAY6AC2A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.674376285 (102h13m40.225846746s) [noise=true , address=77.90.31.17:4776 , piece_id=254SH54AOFXNCTTM6QF3AUN7DYYRE2EKBHDBEUKLHEJMKDBXSVDQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.678052054 (102h13m40.229522505s) [noise=true , address=145.79.227.38:4984 , piece_id=LP425WCAJOPGTKLRDOU6V2M3LWRGZ3DZLF4JL4L5QDHW4V37TOZA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.761077612 (102h13m40.312548063s) [noise=true , address=77.90.31.14:4678 , piece_id=D4HEZCFBVRL5CMXNWRCWXFOA3UMT2NXCMIA2L4ACWQ25O3PU4B4A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.771558030 (102h13m40.323028481s) [noise=true , address=45.91.116.6:4993 , piece_id=B7J3JA5UFDKUW6Q6YACXNQY7EUH7PXYRFUOMMLQKOCWJ6NZ3G3DQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.772042953 (102h13m40.323513414s) [noise=true , address=45.91.116.11:4145 , piece_id=7SLQGSEAL7WCGUQ6TRYWNM6F57TZKH6PNDNVXWQGTZ6LKKVXBONA, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775497421 (102h13m40.326967872s) [noise=true , address=45.91.116.14:4203 , piece_id=SU2IO7NSAAQP4ZEY4QIHXQYOHFFDOSGFQEHFJG44QLIPA37DUA6Q, seg_pos={0 0}, upload=89995]: Done uploading piece. err: <nil> Nov 13 14:17:41.775520495 (102h13m40.326990956s) [seg_pos={0 0}, upload=89995]: Segment reached optimal threshold of 14 pieces. Nov 13 14:17:41.775700267 (102h13m40.327170729s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1D4NHwWuUoNH51rQCv9jdJ4oKKHxMQJ2ZZJfrSZLWYWJJWipWK): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775707361 (102h13m40.327177812s) [noise=true , address=77.90.31.16:4767 , piece_id=A4QMPADSZ2ZGQBOCVQQXKJN65XIEQCHWL2X4KIIKQFTJSNXNED7A, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.775974550 (102h13m40.327445001s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.775982485 (102h13m40.327452936s) [noise=true , address=77.90.31.11:4574 , piece_id=BKX26CT6MB3AA6SMFMVPAJIQAHW35FUUJ6IS24O32675WJTGWHBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776090991 (102h13m40.327561442s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776096271 (102h13m40.327566732s) [noise=true , address=45.91.116.9:4274 , piece_id=K77MUM3I6H3DTNV7RCPSSE7AZH46URTISUPHFC7G4ZABRU7BAWTQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776208986 (102h13m40.327679437s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU): protocol: expected piece hash; context canceled; EOF Nov 13 14:17:41.776214626 (102h13m40.327685087s) [noise=true , address=77.90.31.12:4594 , piece_id=VE65JN4L7ELAEGQFBBQJJCXPOYI5BFQZY5J7WA56LEFUQTS2JJBQ, seg_pos={0 0}, upload=89995]: Long tail cancellation due to optimal threshold Nov 13 14:17:41.776231869 (102h13m40.32770232s) [seg_pos={0 0}, upload=89995]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 13 14:17:41.776317181 (102h13m40.327787632s) [upload=89995]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:17:41.776322282 (102h13m40.327792743s) [upload=89995]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:18:48.053973011 (102h14m46.605443472s) [upload=89995]: Done uploading segments... Nov 13 14:18:48.054067461 (102h14m46.605537912s) [upload=89995]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.863888006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.414476291s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:37:46 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.414498762s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 6.125411541s trace: https://jaeger.dev.storj.io/trace/2a65ce5005cafd1d FAIL (2025-11-13 14:37:44 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.125472128s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.305423752s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:37:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.305452055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.478960273s trace: https://jaeger.dev.storj.io/trace/403ba97a5a1cde7d FAIL (2025-11-13 14:37:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:00.282283372 (102h11m58.833753833s) [upload=89970]: Starting upload Nov 13 14:16:00.282383773 (102h11m58.833854234s) [upload=89970]: Done starting upload Nov 13 14:16:00.282466601 (102h11m58.833937052s) [upload=89970]: Uploading segments... Nov 13 14:16:00.282681841 (102h11m58.834152302s) [upload=89970]: Got next segment. Inline: false Nov 13 14:16:00.282731475 (102h11m58.834201936s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:00.282737627 (102h11m58.834208088s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 14:16:02.313071205 (102h12m0.864541666s) [seg_pos={0 0}, upload=89970]: Begin upload segment... Nov 13 14:16:02.313149905 (102h12m0.864620376s) [seg_pos={0 0}, upload=89970]: Done begin upload segment. Nov 13 14:16:02.313271697 (102h12m0.864742158s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.313324648 (102h12m0.864795129s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314320904 (102h12m0.865791355s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.314744151 (102h12m0.866214642s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315158950 (102h12m0.866629421s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315599659 (102h12m0.867070121s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.315928506 (102h12m0.867398967s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.316477040 (102h12m0.867947501s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317049180 (102h12m0.868519651s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317427380 (102h12m0.868897841s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.317934746 (102h12m0.869405207s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318376678 (102h12m0.869847139s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.318812577 (102h12m0.870283038s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319367303 (102h12m0.870837765s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.319957317 (102h12m0.871427768s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:02.320489491 (102h12m0.871959952s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Uploading piece... Nov 13 14:16:03.991369116 (102h12m2.542839567s) [upload=89970]: Next returned nil segment Nov 13 14:16:03.991386198 (102h12m2.542856649s) [upload=89970]: Waiting for error group managing segments... Nov 13 14:16:04.278532114 (102h12m2.830002575s) [noise=true , address=145.79.227.22:4510 , piece_id=352KKY6FRPZRSXRVBOAMZEBXKWLVHHKV4PBILCEQG7O7QQHCSYBQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.279571643 (102h12m2.831042105s) [noise=true , address=145.79.227.10:4401 , piece_id=ZSHXYHPQVVB3S5VV2TC4N7VJNMH63273HE7TQINN7IRAS3TTBJFA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.289684120 (102h12m2.841154581s) [noise=true , address=145.79.227.8:4329 , piece_id=PT3IKPLS2USVEAQA2KROLOX2JNEYKH77TJ4C5R4KF2GMXXH3I53A, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.300556964 (102h12m2.852027435s) [noise=true , address=145.79.227.30:4762 , piece_id=WVQ7ELVUTR3ACA4LRW2OHY5IHNAC352ZGL4SQB7ZZD2GWZHPAFBA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.301333473 (102h12m2.852803934s) [noise=true , address=145.79.227.28:4727 , piece_id=ISQYCRF3CIEKWWNLT2LJEXKTCP23RAHMIFPUWFHPYTL23R4H5FMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302408349 (102h12m2.853878821s) [noise=true , address=145.79.227.30:4763 , piece_id=QWY35BJ3SXD3D4MDECKNDDKYTKAL7EPVUKRDCEPM7EKBSC5X2RAQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.302631235 (102h12m2.854101716s) [noise=true , address=145.79.227.9:4367 , piece_id=MNC2JVAUM7DVMPK7TRNIAHXTMO53MDXVQLQ54ATHGPCXW4PKJ4HA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.335556962 (102h12m2.887027413s) [noise=true , address=145.79.227.32:4836 , piece_id=XMZC4XBTVGYDLU3JNHL3HIHCQK535VNJ7GG6PIQBGBO7SPIVO4VA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.346910641 (102h12m2.898381102s) [noise=true , address=145.79.227.28:4725 , piece_id=7I2KAFSFRUGPGI2O55IWYCTL5VYGAUF34W5Q6BC4LVKESK5I4TUQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.371810816 (102h12m2.923281277s) [noise=true , address=145.79.227.24:4581 , piece_id=TFI7RYWYVP4RAMS3HLE2VSCJPCECNCMYQZXUXUWI3YG6VVT5VKCA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377236428 (102h12m2.928706889s) [noise=true , address=145.79.227.11:4729 , piece_id=USKPGFKOOEBATKFBCAUW2GBJNSIYBM2NC5H5GNNYJ2OOVDHZDK2Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.377262828 (102h12m2.928733289s) [noise=true , address=145.79.227.14:4839 , piece_id=K5S246U63MSHTE2TN3Q4LVGLY5MALYWDRG47G4M3DAST4AK7J65Q, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383693804 (102h12m2.935164265s) [noise=true , address=145.79.227.14:4837 , piece_id=YDBMWYRBKEZGG6Z4PPKRSSBUENSQDNN7HHLUCD5ZHELKKNGRPROQ, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383710837 (102h12m2.935181288s) [seg_pos={0 0}, upload=89970]: Segment reached optimal threshold of 13 pieces. Nov 13 14:16:04.383822670 (102h12m2.935293121s) [noise=true , address=145.79.227.31:4799 , piece_id=WVXCSA4LG4KGGO54GAYD34HWHBMUMS3K73KQZA7NQBIH7WPHRWIA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: <nil> Nov 13 14:16:04.383922450 (102h12m2.935392911s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53612->145.79.227.31:4798: use of closed network connection Nov 13 14:16:04.383929954 (102h12m2.935400415s) [noise=true , address=145.79.227.31:4798 , piece_id=VFY7BQXJBF5WSP5ZI2HFTMHPCSTLDMY42FSUROZVZWD35MCJ7RZA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384043279 (102h12m2.93551374s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39110->45.91.116.6:4974: use of closed network connection Nov 13 14:16:04.384050112 (102h12m2.935520563s) [noise=true , address=45.91.116.6:4974 , piece_id=EGPVXYQZOSE3CPJQK4GLTJF35JEDZXURYM7NKUHNXEHUWOOCIAMA, seg_pos={0 0}, upload=89970]: Long tail cancellation due to optimal threshold Nov 13 14:16:04.384066143 (102h12m2.935536604s) [seg_pos={0 0}, upload=89970]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Nov 13 14:16:04.384159561 (102h12m2.935630013s) [upload=89970]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 13 14:16:04.384171734 (102h12m2.935642195s) [upload=89970]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:16:04.934815134 (102h12m3.486285595s) [upload=89970]: Done uploading segments... Nov 13 14:16:04.934864327 (102h12m3.486334788s) [upload=89970]: Upload finished. err: <nil> Duration: 6.478991132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.333402087s trace: https://jaeger.dev.storj.io/trace/afd242eeed2927 FAIL (2025-11-13 14:37:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.333432717s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.178411083s trace: https://jaeger.dev.storj.io/trace/d6ec5f1d62f8624 FAIL (2025-11-13 14:37:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 13 14:16:32.932592910 (102h12m31.484063371s) [upload=89979]: Starting upload Nov 13 14:16:32.932874286 (102h12m31.484344767s) [upload=89979]: Done starting upload Nov 13 14:16:32.933042266 (102h12m31.484512717s) [upload=89979]: Uploading segments... Nov 13 14:16:32.933200005 (102h12m31.484670467s) [upload=89979]: Got next segment. Inline: true Nov 13 14:16:32.933208482 (102h12m31.484678943s) [upload=89979]: Waiting for error group managing segments... Nov 13 14:16:32.933309854 (102h12m31.484780316s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 14:16:32.933317860 (102h12m31.484788322s) [upload=89979]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Nov 13 14:16:32.933321948 (102h12m31.484792409s) [upload=89979]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 13 14:17:01.188559179 (102h12m59.74002964s) [upload=89979]: Done uploading segments... Nov 13 14:17:01.188737939 (102h12m59.7402084s) [upload=89979]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.178463774s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)