Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 15.486610432s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:05:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 15.486642151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X......X...................X.........X....X.X......XX.......] duration: 6.049996072s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:05:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 6.050023645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X......X...................X.........X....X.X......XX......] duration: 24.872139272s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:04:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 24.872165463s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.722505463s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:04:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.722534747s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 17.932735876s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 17.932762266s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.300212179s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:03:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.300238799s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 21.556690509s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:03:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 21.556730645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X......X...................X.........X....X.X......XX.....] duration: 5.77193419s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:03:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 5.771961852s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.174170282s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:02:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.174193718s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X......X...................X.........X....X.X......XX....] duration: 5.847345032s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:02:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 5.847371852s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.029071897s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:02:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.029135709s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 28.212714877s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:02:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 28.21276348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X......X...................X.........X....X.X......XX...] duration: 25.676065321s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:02:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 25.676133169s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.223036633s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:01:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.223060128s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X......X...................X.........X....X.X......XX..] duration: 6.017883947s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:01:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 6.017975191s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 21.537615526s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:01:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 21.537641065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.....X......X...................X.........X....X.X......XX.] duration: 6.468709518s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:00:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 6.468724065s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.X................................] duration: 19.884235439s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:00:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 19.88426246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.....X......X...................X.........X....X.X......XX] duration: 30.003607062s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:00:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 30.003618404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration: 25.516232472s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:00:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 25.516263049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.483208946s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:00:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.483239946s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X.....X......X...................X.........X....X.X......X] duration: 30.003803142s trace: https://jaeger.dev.storj.io/trace/7c8949b6d2402ce9 FAIL (2025-12-21 15:00:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21 14:59:48.624450401 (383h30m48.495533436s) [upload=336598]: Starting upload Dec 21 14:59:48.624523650 (383h30m48.495606695s) [upload=336598]: Done starting upload Dec 21 14:59:48.624574006 (383h30m48.495657041s) [upload=336598]: Uploading segments... Dec 21 14:59:48.624755703 (383h30m48.495838738s) [upload=336598]: Got next segment. Inline: false Dec 21 14:59:48.624800177 (383h30m48.495883212s) [upload=336598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:59:48.624804505 (383h30m48.49588756s) [upload=336598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:59:49.884523961 (383h30m49.755606996s) [seg_pos={0 0}, upload=336598]: Begin upload segment... Dec 21 14:59:49.884571251 (383h30m49.755654296s) [seg_pos={0 0}, upload=336598]: Done begin upload segment. Dec 21 14:59:49.884589775 (383h30m49.75567281s) [noise=true , address=145.79.227.39:4014 , piece_id=T35NVQLDBTEI7BYZVO3CZRFGA7W3J45CRODRK4HOATMQ24AKHN6Q, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.884635423 (383h30m49.755718458s) [noise=true , address=145.79.227.36:4518 , piece_id=K3YG6KCCOZVQ54O7ORPINBPC2ZYUQ6C2QMW2I4Y7ZB5ON2AC6ZCA, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.884992222 (383h30m49.756075257s) [noise=true , address=145.79.227.17:4972 , piece_id=AXYGGJQ3WPI2P5ADSGW46R6IZ2P26TAGJFZUQYJFQG2LZOMKTTGA, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.885260854 (383h30m49.756343889s) [noise=true , address=145.79.227.52:4445 , piece_id=ATE3ZFHTNIDOTTZSC6Y4WIUZXUMKW7252AL3CLHSQAEECFTFBK3Q, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.885527461 (383h30m49.756610496s) [noise=true , address=145.79.227.2:4114 , piece_id=3ZYISPXCO6DBCBUGDEXGMFZBC37GG2HY2GMM662MP2LZM3UGFLYQ, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.886122163 (383h30m49.757205208s) [noise=true , address=45.91.116.12:4225 , piece_id=H6XZ7UU3H6WRMJBBVVYJLWAQJEHMS5VRH2XHKEF2VOIXLGRPRZUQ, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.886516154 (383h30m49.757599189s) [noise=true , address=145.79.227.31:4797 , piece_id=HWGZNTDRUX75GCCLB6OZD2TY6KOA4H3BJLQEXAB3VNBYKGD76KXA, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.887311529 (383h30m49.758394564s) [noise=true , address=145.79.227.51:4409 , piece_id=DPONEKLJITIK3TDWRCA5EMWEUXBHZS4D3JFDC3BFRGV2F6RCRZHQ, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.888480725 (383h30m49.75956376s) [noise=true , address=145.79.227.3:4151 , piece_id=SZJYDDKP4R4WY7JOG56O5BMQ2PXUGJB7VZKPHCNULI7XPM6KYPQQ, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.889517860 (383h30m49.760600905s) [noise=true , address=145.79.227.46:4603 , piece_id=V2YUZUJMHIHLWIO7PS5OEQYSHKHSPBYNAXSOF2N4LNWJQLUESI4Q, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.890724347 (383h30m49.761807382s) [noise=true , address=45.91.116.9:4261 , piece_id=XHAUPPBHKEGKVJU2UCYOSMYVKCD4HC6GABO7ZUJETVJ6PV55J5LA, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.892234623 (383h30m49.763317658s) [noise=true , address=145.79.227.37:4565 , piece_id=L4CN577FK5T5R7VUUZGBCMXVCLIO44B465UOFQIX3TRADH6FGGSA, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.893335849 (383h30m49.764418884s) [noise=true , address=145.79.227.27:4691 , piece_id=ZBOQ6TWP42AA3A2IQ4JW5CEFLYFGVFS7PWYZO4DCKIIMNB4P73LQ, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.894495357 (383h30m49.765578392s) [noise=true , address=145.79.227.10:4401 , piece_id=VBSSXLTBTBQC6H2GDW3MP3EQZDZVM7AUFCSZWWXIL5Z4J3AZTKQQ, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.895229546 (383h30m49.766312611s) [noise=true , address=145.79.227.42:4122 , piece_id=HOIAYBFIXF5WP3OKBIENFV2UALDJV4AWWJJEDZL7ZVYB4JGGV2KQ, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:49.896126523 (383h30m49.767209558s) [noise=true , address=145.79.227.49:4338 , piece_id=6KCV6YD4HSVPGZ2QVOKVS5XL6OLMUEIXNVBGURZYQELMHONBT5QQ, seg_pos={0 0}, upload=336598]: Uploading piece... Dec 21 14:59:51.748913012 (383h30m51.619996047s) [upload=336598]: Next returned nil segment Dec 21 14:59:51.748931687 (383h30m51.620014722s) [upload=336598]: Waiting for error group managing segments... Dec 21 14:59:52.055560692 (383h30m51.926643727s) [noise=true , address=145.79.227.51:4409 , piece_id=DPONEKLJITIK3TDWRCA5EMWEUXBHZS4D3JFDC3BFRGV2F6RCRZHQ, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.056436419 (383h30m51.927519464s) [noise=true , address=145.79.227.2:4114 , piece_id=3ZYISPXCO6DBCBUGDEXGMFZBC37GG2HY2GMM662MP2LZM3UGFLYQ, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.056487898 (383h30m51.927570933s) [noise=true , address=145.79.227.36:4518 , piece_id=K3YG6KCCOZVQ54O7ORPINBPC2ZYUQ6C2QMW2I4Y7ZB5ON2AC6ZCA, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.066958475 (383h30m51.93804151s) [noise=true , address=145.79.227.46:4603 , piece_id=V2YUZUJMHIHLWIO7PS5OEQYSHKHSPBYNAXSOF2N4LNWJQLUESI4Q, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.069355439 (383h30m51.940438474s) [noise=true , address=145.79.227.42:4122 , piece_id=HOIAYBFIXF5WP3OKBIENFV2UALDJV4AWWJJEDZL7ZVYB4JGGV2KQ, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.100696597 (383h30m51.971779632s) [noise=true , address=145.79.227.52:4445 , piece_id=ATE3ZFHTNIDOTTZSC6Y4WIUZXUMKW7252AL3CLHSQAEECFTFBK3Q, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.111373628 (383h30m51.982456693s) [noise=true , address=145.79.227.37:4565 , piece_id=L4CN577FK5T5R7VUUZGBCMXVCLIO44B465UOFQIX3TRADH6FGGSA, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.114970757 (383h30m51.986053792s) [noise=true , address=145.79.227.27:4691 , piece_id=ZBOQ6TWP42AA3A2IQ4JW5CEFLYFGVFS7PWYZO4DCKIIMNB4P73LQ, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.117986619 (383h30m51.989069654s) [noise=true , address=145.79.227.17:4972 , piece_id=AXYGGJQ3WPI2P5ADSGW46R6IZ2P26TAGJFZUQYJFQG2LZOMKTTGA, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.127601167 (383h30m51.998684202s) [noise=true , address=145.79.227.39:4014 , piece_id=T35NVQLDBTEI7BYZVO3CZRFGA7W3J45CRODRK4HOATMQ24AKHN6Q, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.131918458 (383h30m52.003001493s) [noise=true , address=145.79.227.10:4401 , piece_id=VBSSXLTBTBQC6H2GDW3MP3EQZDZVM7AUFCSZWWXIL5Z4J3AZTKQQ, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.134416615 (383h30m52.00549965s) [noise=true , address=145.79.227.49:4338 , piece_id=6KCV6YD4HSVPGZ2QVOKVS5XL6OLMUEIXNVBGURZYQELMHONBT5QQ, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.139793613 (383h30m52.010876658s) [noise=true , address=145.79.227.31:4797 , piece_id=HWGZNTDRUX75GCCLB6OZD2TY6KOA4H3BJLQEXAB3VNBYKGD76KXA, seg_pos={0 0}, upload=336598]: Done uploading piece. err: <nil> Dec 21 14:59:52.139933530 (383h30m52.011016565s) [seg_pos={0 0}, upload=336598]: Segment reached optimal threshold of 13 pieces. Dec 21 14:59:52.140112540 (383h30m52.011195575s) [noise=true , address=145.79.227.3:4151 , piece_id=SZJYDDKP4R4WY7JOG56O5BMQ2PXUGJB7VZKPHCNULI7XPM6KYPQQ, seg_pos={0 0}, upload=336598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:50616->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:50616->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50616->145.79.227.3:4151: use of closed network connection Dec 21 14:59:52.140119023 (383h30m52.011202058s) [noise=true , address=145.79.227.3:4151 , piece_id=SZJYDDKP4R4WY7JOG56O5BMQ2PXUGJB7VZKPHCNULI7XPM6KYPQQ, seg_pos={0 0}, upload=336598]: Long tail cancellation due to optimal threshold Dec 21 14:59:52.140275000 (383h30m52.011358035s) [noise=true , address=45.91.116.12:4225 , piece_id=H6XZ7UU3H6WRMJBBVVYJLWAQJEHMS5VRH2XHKEF2VOIXLGRPRZUQ, seg_pos={0 0}, upload=336598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 14:59:52.140280661 (383h30m52.011363696s) [noise=true , address=45.91.116.12:4225 , piece_id=H6XZ7UU3H6WRMJBBVVYJLWAQJEHMS5VRH2XHKEF2VOIXLGRPRZUQ, seg_pos={0 0}, upload=336598]: Long tail cancellation due to optimal threshold Dec 21 14:59:52.140397694 (383h30m52.011480739s) [noise=true , address=45.91.116.9:4261 , piece_id=XHAUPPBHKEGKVJU2UCYOSMYVKCD4HC6GABO7ZUJETVJ6PV55J5LA, seg_pos={0 0}, upload=336598]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 21 14:59:52.140404617 (383h30m52.011487652s) [noise=true , address=45.91.116.9:4261 , piece_id=XHAUPPBHKEGKVJU2UCYOSMYVKCD4HC6GABO7ZUJETVJ6PV55J5LA, seg_pos={0 0}, upload=336598]: Long tail cancellation due to optimal threshold Dec 21 14:59:52.140419946 (383h30m52.011502981s) [seg_pos={0 0}, upload=336598]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:59:52.140487565 (383h30m52.0115706s) [upload=336598]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:59:52.140491933 (383h30m52.011574978s) [upload=336598]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:59:52.454435406 (383h30m52.325518451s) [upload=336598]: Done uploading segments... Dec 21 14:59:52.454462938 (383h30m52.325545983s) [upload=336598]: Upload finished. err: <nil> Duration: 30.003825125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 21.477467122s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:59:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 21.47748191s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.427075275s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 14:59:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.427110202s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 20.487993458s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:59:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 20.488017743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.947760088s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 14:59:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.94778801s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X.....X......X...................X.........X....X.X......] duration: 6.072595553s trace: https://jaeger.dev.storj.io/trace/42348012034fa4e2 FAIL (2025-12-21 14:58:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (383h25m26.575797232s) [upload=336515]: Done starting upload Dec 21 14:54:26.704921763 (383h25m26.576004808s) [upload=336515]: Uploading segments... Dec 21 14:54:26.705037573 (383h25m26.576120628s) [upload=336515]: Got next segment. Inline: false Dec 21 14:54:26.705135860 (383h25m26.576218905s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:54:26.705143185 (383h25m26.57622624s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:54:26.951288109 (383h25m26.822371144s) [seg_pos={0 0}, upload=336515]: Begin upload segment... Dec 21 14:54:26.951327775 (383h25m26.82241081s) [seg_pos={0 0}, upload=336515]: Done begin upload segment. Dec 21 14:54:26.951354356 (383h25m26.822437401s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951425371 (383h25m26.822508406s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951974195 (383h25m26.82305724s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952303513 (383h25m26.823386558s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952624173 (383h25m26.823707208s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953005048 (383h25m26.824088093s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953344575 (383h25m26.82442762s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953712665 (383h25m26.824795711s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.954280878 (383h25m26.825363913s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.955858462 (383h25m26.826941497s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.956863175 (383h25m26.82794621s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.958434196 (383h25m26.829517231s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.959857205 (383h25m26.83094024s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.961297538 (383h25m26.832380574s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.963131972 (383h25m26.834215007s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.964910128 (383h25m26.835993163s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:28.667195901 (383h25m28.538278936s) [upload=336515]: Next returned nil segment Dec 21 14:54:28.667228573 (383h25m28.538311608s) [upload=336515]: Waiting for error group managing segments... Dec 21 14:54:28.941547916 (383h25m28.812630951s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.953479044 (383h25m28.824562079s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.959358690 (383h25m28.830441725s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.964430627 (383h25m28.835513662s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.966894058 (383h25m28.837977113s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.000141596 (383h25m28.871224651s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.004395286 (383h25m28.875478321s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.020432914 (383h25m28.891515949s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.192391568 (383h25m29.063474613s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.214089984 (383h25m29.085173029s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.215340044 (383h25m29.086423089s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.292588807 (383h25m29.163671862s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315054185 (383h25m29.18613722s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315108948 (383h25m29.186191984s) [seg_pos={0 0}, upload=336515]: Segment reached optimal threshold of 13 pieces. Dec 21 14:54:29.315341381 (383h25m29.186424416s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection Dec 21 14:54:29.315347924 (383h25m29.186430959s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315449487 (383h25m29.186532522s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:48842->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 14:54:29.315459396 (383h25m29.186542441s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315605514 (383h25m29.186688559s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 21 14:54:29.315612067 (383h25m29.186695112s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315628378 (383h25m29.186711423s) [seg_pos={0 0}, upload=336515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:54:29.315706686 (383h25m29.186789732s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:54:29.315720012 (383h25m29.186803058s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:54:29.617901108 (383h25m29.488984143s) [upload=336515]: Done uploading segments... Dec 21 14:54:29.617922258 (383h25m29.489005294s) [upload=336515]: Upload finished. err: <nil> Duration: 6.072622564s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.624603684s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 14:58:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.624628802s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 20.443012218s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:58:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 20.443035292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X..X.....X......X...................X.........X....X.X.....] duration: 6.00232278s trace: https://jaeger.dev.storj.io/trace/42348012034fa4e2 FAIL (2025-12-21 14:58:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (383h25m26.575797232s) [upload=336515]: Done starting upload Dec 21 14:54:26.704921763 (383h25m26.576004808s) [upload=336515]: Uploading segments... Dec 21 14:54:26.705037573 (383h25m26.576120628s) [upload=336515]: Got next segment. Inline: false Dec 21 14:54:26.705135860 (383h25m26.576218905s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:54:26.705143185 (383h25m26.57622624s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:54:26.951288109 (383h25m26.822371144s) [seg_pos={0 0}, upload=336515]: Begin upload segment... Dec 21 14:54:26.951327775 (383h25m26.82241081s) [seg_pos={0 0}, upload=336515]: Done begin upload segment. Dec 21 14:54:26.951354356 (383h25m26.822437401s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951425371 (383h25m26.822508406s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951974195 (383h25m26.82305724s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952303513 (383h25m26.823386558s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952624173 (383h25m26.823707208s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953005048 (383h25m26.824088093s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953344575 (383h25m26.82442762s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953712665 (383h25m26.824795711s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.954280878 (383h25m26.825363913s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.955858462 (383h25m26.826941497s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.956863175 (383h25m26.82794621s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.958434196 (383h25m26.829517231s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.959857205 (383h25m26.83094024s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.961297538 (383h25m26.832380574s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.963131972 (383h25m26.834215007s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.964910128 (383h25m26.835993163s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:28.667195901 (383h25m28.538278936s) [upload=336515]: Next returned nil segment Dec 21 14:54:28.667228573 (383h25m28.538311608s) [upload=336515]: Waiting for error group managing segments... Dec 21 14:54:28.941547916 (383h25m28.812630951s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.953479044 (383h25m28.824562079s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.959358690 (383h25m28.830441725s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.964430627 (383h25m28.835513662s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.966894058 (383h25m28.837977113s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.000141596 (383h25m28.871224651s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.004395286 (383h25m28.875478321s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.020432914 (383h25m28.891515949s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.192391568 (383h25m29.063474613s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.214089984 (383h25m29.085173029s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.215340044 (383h25m29.086423089s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.292588807 (383h25m29.163671862s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315054185 (383h25m29.18613722s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315108948 (383h25m29.186191984s) [seg_pos={0 0}, upload=336515]: Segment reached optimal threshold of 13 pieces. Dec 21 14:54:29.315341381 (383h25m29.186424416s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection Dec 21 14:54:29.315347924 (383h25m29.186430959s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315449487 (383h25m29.186532522s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:48842->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 14:54:29.315459396 (383h25m29.186542441s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315605514 (383h25m29.186688559s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 21 14:54:29.315612067 (383h25m29.186695112s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315628378 (383h25m29.186711423s) [seg_pos={0 0}, upload=336515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:54:29.315706686 (383h25m29.186789732s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:54:29.315720012 (383h25m29.186803058s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:54:29.617901108 (383h25m29.488984143s) [upload=336515]: Done uploading segments... Dec 21 14:54:29.617922258 (383h25m29.489005294s) [upload=336515]: Upload finished. err: <nil> Duration: 6.002349562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 23.0016069s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:57:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 23.001629863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X..X.....X......X...................X.........X....X.X....] duration: 5.937043696s trace: https://jaeger.dev.storj.io/trace/42348012034fa4e2 FAIL (2025-12-21 14:57:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (383h25m26.575797232s) [upload=336515]: Done starting upload Dec 21 14:54:26.704921763 (383h25m26.576004808s) [upload=336515]: Uploading segments... Dec 21 14:54:26.705037573 (383h25m26.576120628s) [upload=336515]: Got next segment. Inline: false Dec 21 14:54:26.705135860 (383h25m26.576218905s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:54:26.705143185 (383h25m26.57622624s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:54:26.951288109 (383h25m26.822371144s) [seg_pos={0 0}, upload=336515]: Begin upload segment... Dec 21 14:54:26.951327775 (383h25m26.82241081s) [seg_pos={0 0}, upload=336515]: Done begin upload segment. Dec 21 14:54:26.951354356 (383h25m26.822437401s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951425371 (383h25m26.822508406s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951974195 (383h25m26.82305724s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952303513 (383h25m26.823386558s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952624173 (383h25m26.823707208s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953005048 (383h25m26.824088093s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953344575 (383h25m26.82442762s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953712665 (383h25m26.824795711s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.954280878 (383h25m26.825363913s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.955858462 (383h25m26.826941497s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.956863175 (383h25m26.82794621s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.958434196 (383h25m26.829517231s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.959857205 (383h25m26.83094024s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.961297538 (383h25m26.832380574s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.963131972 (383h25m26.834215007s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.964910128 (383h25m26.835993163s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:28.667195901 (383h25m28.538278936s) [upload=336515]: Next returned nil segment Dec 21 14:54:28.667228573 (383h25m28.538311608s) [upload=336515]: Waiting for error group managing segments... Dec 21 14:54:28.941547916 (383h25m28.812630951s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.953479044 (383h25m28.824562079s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.959358690 (383h25m28.830441725s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.964430627 (383h25m28.835513662s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.966894058 (383h25m28.837977113s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.000141596 (383h25m28.871224651s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.004395286 (383h25m28.875478321s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.020432914 (383h25m28.891515949s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.192391568 (383h25m29.063474613s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.214089984 (383h25m29.085173029s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.215340044 (383h25m29.086423089s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.292588807 (383h25m29.163671862s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315054185 (383h25m29.18613722s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315108948 (383h25m29.186191984s) [seg_pos={0 0}, upload=336515]: Segment reached optimal threshold of 13 pieces. Dec 21 14:54:29.315341381 (383h25m29.186424416s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection Dec 21 14:54:29.315347924 (383h25m29.186430959s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315449487 (383h25m29.186532522s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:48842->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 14:54:29.315459396 (383h25m29.186542441s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315605514 (383h25m29.186688559s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 21 14:54:29.315612067 (383h25m29.186695112s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315628378 (383h25m29.186711423s) [seg_pos={0 0}, upload=336515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:54:29.315706686 (383h25m29.186789732s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:54:29.315720012 (383h25m29.186803058s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:54:29.617901108 (383h25m29.488984143s) [upload=336515]: Done uploading segments... Dec 21 14:54:29.617922258 (383h25m29.489005294s) [upload=336515]: Upload finished. err: <nil> Duration: 5.93707234s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.249462194s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 14:57:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.249492934s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 15.155312559s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:57:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 15.155340111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X..X.....X......X...................X.........X....X.X...] duration: 7.604884176s trace: https://jaeger.dev.storj.io/trace/42348012034fa4e2 FAIL (2025-12-21 14:57:14 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (383h25m26.575797232s) [upload=336515]: Done starting upload Dec 21 14:54:26.704921763 (383h25m26.576004808s) [upload=336515]: Uploading segments... Dec 21 14:54:26.705037573 (383h25m26.576120628s) [upload=336515]: Got next segment. Inline: false Dec 21 14:54:26.705135860 (383h25m26.576218905s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:54:26.705143185 (383h25m26.57622624s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:54:26.951288109 (383h25m26.822371144s) [seg_pos={0 0}, upload=336515]: Begin upload segment... Dec 21 14:54:26.951327775 (383h25m26.82241081s) [seg_pos={0 0}, upload=336515]: Done begin upload segment. Dec 21 14:54:26.951354356 (383h25m26.822437401s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951425371 (383h25m26.822508406s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951974195 (383h25m26.82305724s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952303513 (383h25m26.823386558s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952624173 (383h25m26.823707208s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953005048 (383h25m26.824088093s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953344575 (383h25m26.82442762s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953712665 (383h25m26.824795711s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.954280878 (383h25m26.825363913s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.955858462 (383h25m26.826941497s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.956863175 (383h25m26.82794621s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.958434196 (383h25m26.829517231s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.959857205 (383h25m26.83094024s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.961297538 (383h25m26.832380574s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.963131972 (383h25m26.834215007s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.964910128 (383h25m26.835993163s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:28.667195901 (383h25m28.538278936s) [upload=336515]: Next returned nil segment Dec 21 14:54:28.667228573 (383h25m28.538311608s) [upload=336515]: Waiting for error group managing segments... Dec 21 14:54:28.941547916 (383h25m28.812630951s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.953479044 (383h25m28.824562079s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.959358690 (383h25m28.830441725s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.964430627 (383h25m28.835513662s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.966894058 (383h25m28.837977113s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.000141596 (383h25m28.871224651s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.004395286 (383h25m28.875478321s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.020432914 (383h25m28.891515949s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.192391568 (383h25m29.063474613s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.214089984 (383h25m29.085173029s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.215340044 (383h25m29.086423089s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.292588807 (383h25m29.163671862s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315054185 (383h25m29.18613722s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315108948 (383h25m29.186191984s) [seg_pos={0 0}, upload=336515]: Segment reached optimal threshold of 13 pieces. Dec 21 14:54:29.315341381 (383h25m29.186424416s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection Dec 21 14:54:29.315347924 (383h25m29.186430959s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315449487 (383h25m29.186532522s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:48842->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 14:54:29.315459396 (383h25m29.186542441s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315605514 (383h25m29.186688559s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 21 14:54:29.315612067 (383h25m29.186695112s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315628378 (383h25m29.186711423s) [seg_pos={0 0}, upload=336515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:54:29.315706686 (383h25m29.186789732s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:54:29.315720012 (383h25m29.186803058s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:54:29.617901108 (383h25m29.488984143s) [upload=336515]: Done uploading segments... Dec 21 14:54:29.617922258 (383h25m29.489005294s) [upload=336515]: Upload finished. err: <nil> Duration: 7.60492283s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.140849766s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 14:56:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.140875545s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 24.840548466s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:56:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 24.840578945s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.60971696s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 14:56:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.609744614s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X..X.....X......X...................X.........X....X.X..] duration: 5.663479778s trace: https://jaeger.dev.storj.io/trace/42348012034fa4e2 FAIL (2025-12-21 14:56:24 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (383h25m26.575797232s) [upload=336515]: Done starting upload Dec 21 14:54:26.704921763 (383h25m26.576004808s) [upload=336515]: Uploading segments... Dec 21 14:54:26.705037573 (383h25m26.576120628s) [upload=336515]: Got next segment. Inline: false Dec 21 14:54:26.705135860 (383h25m26.576218905s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:54:26.705143185 (383h25m26.57622624s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:54:26.951288109 (383h25m26.822371144s) [seg_pos={0 0}, upload=336515]: Begin upload segment... Dec 21 14:54:26.951327775 (383h25m26.82241081s) [seg_pos={0 0}, upload=336515]: Done begin upload segment. Dec 21 14:54:26.951354356 (383h25m26.822437401s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951425371 (383h25m26.822508406s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951974195 (383h25m26.82305724s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952303513 (383h25m26.823386558s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952624173 (383h25m26.823707208s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953005048 (383h25m26.824088093s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953344575 (383h25m26.82442762s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953712665 (383h25m26.824795711s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.954280878 (383h25m26.825363913s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.955858462 (383h25m26.826941497s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.956863175 (383h25m26.82794621s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.958434196 (383h25m26.829517231s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.959857205 (383h25m26.83094024s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.961297538 (383h25m26.832380574s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.963131972 (383h25m26.834215007s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.964910128 (383h25m26.835993163s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:28.667195901 (383h25m28.538278936s) [upload=336515]: Next returned nil segment Dec 21 14:54:28.667228573 (383h25m28.538311608s) [upload=336515]: Waiting for error group managing segments... Dec 21 14:54:28.941547916 (383h25m28.812630951s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.953479044 (383h25m28.824562079s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.959358690 (383h25m28.830441725s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.964430627 (383h25m28.835513662s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.966894058 (383h25m28.837977113s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.000141596 (383h25m28.871224651s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.004395286 (383h25m28.875478321s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.020432914 (383h25m28.891515949s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.192391568 (383h25m29.063474613s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.214089984 (383h25m29.085173029s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.215340044 (383h25m29.086423089s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.292588807 (383h25m29.163671862s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315054185 (383h25m29.18613722s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315108948 (383h25m29.186191984s) [seg_pos={0 0}, upload=336515]: Segment reached optimal threshold of 13 pieces. Dec 21 14:54:29.315341381 (383h25m29.186424416s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection Dec 21 14:54:29.315347924 (383h25m29.186430959s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315449487 (383h25m29.186532522s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:48842->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 14:54:29.315459396 (383h25m29.186542441s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315605514 (383h25m29.186688559s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 21 14:54:29.315612067 (383h25m29.186695112s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315628378 (383h25m29.186711423s) [seg_pos={0 0}, upload=336515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:54:29.315706686 (383h25m29.186789732s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:54:29.315720012 (383h25m29.186803058s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:54:29.617901108 (383h25m29.488984143s) [upload=336515]: Done uploading segments... Dec 21 14:54:29.617922258 (383h25m29.489005294s) [upload=336515]: Upload finished. err: <nil> Duration: 5.663508822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 18.984817065s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:55:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 18.984842233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.121323816s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 14:55:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.121349557s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X..X.....X......X...................X.........X....X.X.] duration: 7.822645813s trace: https://jaeger.dev.storj.io/trace/42348012034fa4e2 FAIL (2025-12-21 14:55:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (383h25m26.575797232s) [upload=336515]: Done starting upload Dec 21 14:54:26.704921763 (383h25m26.576004808s) [upload=336515]: Uploading segments... Dec 21 14:54:26.705037573 (383h25m26.576120628s) [upload=336515]: Got next segment. Inline: false Dec 21 14:54:26.705135860 (383h25m26.576218905s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:54:26.705143185 (383h25m26.57622624s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:54:26.951288109 (383h25m26.822371144s) [seg_pos={0 0}, upload=336515]: Begin upload segment... Dec 21 14:54:26.951327775 (383h25m26.82241081s) [seg_pos={0 0}, upload=336515]: Done begin upload segment. Dec 21 14:54:26.951354356 (383h25m26.822437401s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951425371 (383h25m26.822508406s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951974195 (383h25m26.82305724s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952303513 (383h25m26.823386558s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952624173 (383h25m26.823707208s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953005048 (383h25m26.824088093s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953344575 (383h25m26.82442762s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953712665 (383h25m26.824795711s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.954280878 (383h25m26.825363913s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.955858462 (383h25m26.826941497s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.956863175 (383h25m26.82794621s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.958434196 (383h25m26.829517231s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.959857205 (383h25m26.83094024s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.961297538 (383h25m26.832380574s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.963131972 (383h25m26.834215007s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.964910128 (383h25m26.835993163s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:28.667195901 (383h25m28.538278936s) [upload=336515]: Next returned nil segment Dec 21 14:54:28.667228573 (383h25m28.538311608s) [upload=336515]: Waiting for error group managing segments... Dec 21 14:54:28.941547916 (383h25m28.812630951s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.953479044 (383h25m28.824562079s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.959358690 (383h25m28.830441725s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.964430627 (383h25m28.835513662s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.966894058 (383h25m28.837977113s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.000141596 (383h25m28.871224651s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.004395286 (383h25m28.875478321s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.020432914 (383h25m28.891515949s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.192391568 (383h25m29.063474613s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.214089984 (383h25m29.085173029s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.215340044 (383h25m29.086423089s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.292588807 (383h25m29.163671862s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315054185 (383h25m29.18613722s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315108948 (383h25m29.186191984s) [seg_pos={0 0}, upload=336515]: Segment reached optimal threshold of 13 pieces. Dec 21 14:54:29.315341381 (383h25m29.186424416s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection Dec 21 14:54:29.315347924 (383h25m29.186430959s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315449487 (383h25m29.186532522s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:48842->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 14:54:29.315459396 (383h25m29.186542441s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315605514 (383h25m29.186688559s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 21 14:54:29.315612067 (383h25m29.186695112s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315628378 (383h25m29.186711423s) [seg_pos={0 0}, upload=336515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:54:29.315706686 (383h25m29.186789732s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:54:29.315720012 (383h25m29.186803058s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:54:29.617901108 (383h25m29.488984143s) [upload=336515]: Done uploading segments... Dec 21 14:54:29.617922258 (383h25m29.489005294s) [upload=336515]: Upload finished. err: <nil> Duration: 7.822672824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X..X.....X......X...................X.........X....X.X] duration: 30.001472836s trace: https://jaeger.dev.storj.io/trace/42348012034fa4e2 FAIL (2025-12-21 14:54:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (383h25m26.575797232s) [upload=336515]: Done starting upload Dec 21 14:54:26.704921763 (383h25m26.576004808s) [upload=336515]: Uploading segments... Dec 21 14:54:26.705037573 (383h25m26.576120628s) [upload=336515]: Got next segment. Inline: false Dec 21 14:54:26.705135860 (383h25m26.576218905s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 14:54:26.705143185 (383h25m26.57622624s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:54:26.951288109 (383h25m26.822371144s) [seg_pos={0 0}, upload=336515]: Begin upload segment... Dec 21 14:54:26.951327775 (383h25m26.82241081s) [seg_pos={0 0}, upload=336515]: Done begin upload segment. Dec 21 14:54:26.951354356 (383h25m26.822437401s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951425371 (383h25m26.822508406s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.951974195 (383h25m26.82305724s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952303513 (383h25m26.823386558s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.952624173 (383h25m26.823707208s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953005048 (383h25m26.824088093s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953344575 (383h25m26.82442762s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.953712665 (383h25m26.824795711s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.954280878 (383h25m26.825363913s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.955858462 (383h25m26.826941497s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.956863175 (383h25m26.82794621s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.958434196 (383h25m26.829517231s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.959857205 (383h25m26.83094024s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.961297538 (383h25m26.832380574s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.963131972 (383h25m26.834215007s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:26.964910128 (383h25m26.835993163s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Uploading piece... Dec 21 14:54:28.667195901 (383h25m28.538278936s) [upload=336515]: Next returned nil segment Dec 21 14:54:28.667228573 (383h25m28.538311608s) [upload=336515]: Waiting for error group managing segments... Dec 21 14:54:28.941547916 (383h25m28.812630951s) [noise=true , address=145.79.227.42:4121 , piece_id=K3Z2EKGAF2G6ZUNUQXTGL4LRTCFU6MW7VC2P3PBTH6NTQPHO6L4Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.953479044 (383h25m28.824562079s) [noise=true , address=145.79.227.37:4566 , piece_id=ZDTJQZE656UID5PPEY6NRZSTBT5T5NVSVBDWGMP256DXPWK6BEVA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.959358690 (383h25m28.830441725s) [noise=true , address=145.79.227.38:4979 , piece_id=AEOCAR7XTEAD6ITTYG7UH3IUG627KQSPK5DKFCRYFW7IMCUCICEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.964430627 (383h25m28.835513662s) [noise=true , address=145.79.227.46:4603 , piece_id=ZONTWNC3PTUFP5IVT3ABAJQVBAYNLZ677K5W3ZFZMXBOEEERQ2NA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:28.966894058 (383h25m28.837977113s) [noise=true , address=145.79.227.2:4114 , piece_id=EGBHSHHWBUUH2RPYCMSZWWDI7I6ZYEJQZU46M4FQADQH3J57QLEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.000141596 (383h25m28.871224651s) [noise=true , address=145.79.227.11:4731 , piece_id=GT2CVECVMASXU463ORPJ7LSL7AFFLGA54YISLR6LOQJGG22OKTIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.004395286 (383h25m28.875478321s) [noise=true , address=145.79.227.29:4941 , piece_id=TEWC4ANCPUZBITUMTTEHFTMX465D6S5SXJ4XPNLRQBHLUOL46HSQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.020432914 (383h25m28.891515949s) [noise=true , address=145.79.227.34:4905 , piece_id=WKHUFIVB4IENJK5URJOTA5P635VL3RPMKV2VM25FLZTHBA5M5MEQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.192391568 (383h25m29.063474613s) [noise=true , address=45.91.116.9:4261 , piece_id=YKEWE4VB5MBTMGOEDSPO2XDHFZTPHQGODS2MOZS4O5MKLCX4BV5Q, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.214089984 (383h25m29.085173029s) [noise=true , address=45.91.116.14:4192 , piece_id=NDVW6QKMM2YXDXWWHEFYPSFH535RDQUY3WZBFYP7QGTQULCAYA4A, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.215340044 (383h25m29.086423089s) [noise=true , address=45.91.116.10:4081 , piece_id=EVRQ3X7YVMZNQCTUMH5LQZVWMW7MFR7JDOMDJVGPVFIPET3TVWDA, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.292588807 (383h25m29.163671862s) [noise=true , address=77.90.31.16:4736 , piece_id=DZN3UZGIOLMYMBCDV5NCUQUX4NE26GGQ3SN7DNLFAMSOMNLK2IIQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315054185 (383h25m29.18613722s) [noise=true , address=45.91.116.5:4298 , piece_id=VSEU3IH4IX4ZKODKWQSFTX6XUNMS3H2BHGD5XIC5NZTTC63N42WQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: <nil> Dec 21 14:54:29.315108948 (383h25m29.186191984s) [seg_pos={0 0}, upload=336515]: Segment reached optimal threshold of 13 pieces. Dec 21 14:54:29.315341381 (383h25m29.186424416s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50496->45.91.116.11:4119: use of closed network connection Dec 21 14:54:29.315347924 (383h25m29.186430959s) [noise=true , address=45.91.116.11:4119 , piece_id=ECZFEALZMKIOJKYYXJXQLOVIFVWXLROUN6ET4N5YLVRVD57MKDWQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315449487 (383h25m29.186532522s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: read tcp 188.166.225.170:48842->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 14:54:29.315459396 (383h25m29.186542441s) [noise=true , address=77.90.31.10:4517 , piece_id=2JOBFFQSEK2UH2QRKT5UOGKAXIFAMHMPCDUUO44MR3HVZEEOM6YQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315605514 (383h25m29.186688559s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: expected piece hash; context canceled; EOF Dec 21 14:54:29.315612067 (383h25m29.186695112s) [noise=true , address=145.79.227.28:4727 , piece_id=ACG4XPBMSBMN2JDTUQ225W5VEGKTLNZUBUGEUFDOQQEBA3RT23OQ, seg_pos={0 0}, upload=336515]: Long tail cancellation due to optimal threshold Dec 21 14:54:29.315628378 (383h25m29.186711423s) [seg_pos={0 0}, upload=336515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:54:29.315706686 (383h25m29.186789732s) [upload=336515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:54:29.315720012 (383h25m29.186803058s) [upload=336515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:54:29.617901108 (383h25m29.488984143s) [upload=336515]: Done uploading segments... Dec 21 14:54:29.617922258 (383h25m29.489005294s) [upload=336515]: Upload finished. err: <nil> Duration: 30.001491973s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 22.70125052s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:54:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 22.70127151s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.096098026s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 14:54:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.096136389s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 26.401299489s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:54:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 26.401316942s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.86705685s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 14:54:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.867085934s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 31.116121381s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 14:53:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 31.116142131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X..X.....X......X...................X.........X....X.] duration: 5.869111024s trace: https://jaeger.dev.storj.io/trace/4c3c6311ccf71a70 FAIL (2025-12-21 14:53:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 2UDMPQ3XM3P7NISLAA74YVAZMRQUKYOULMJHMP5YHDRNJ5WDUYWQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: LCP2YLCB6NZEQHP6UNQTHQ4GHDM4FT7SVWBRHISDIQFR6QLW3A7A) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: GNWN2W5VCM6AE24AQK2WNSE6DRITBXE2Y2BO2JVSAQZOQIBGO7MA) context deadline exceeded ======LOGS====== 57.040323002 (383h23m56.911406047s) [upload=336495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:52:57.344790141 (383h23m57.215873186s) [seg_pos={0 0}, upload=336495]: Begin upload segment... Dec 21 14:52:57.344861838 (383h23m57.215967947s) [seg_pos={0 0}, upload=336495]: Done begin upload segment. Dec 21 14:52:57.344894730 (383h23m57.215977765s) [noise=true , address=45.91.116.7:4009 , piece_id=XR4FURKTH7PLUFFFT4ZK4RFPNUWMR6O7FKMLDURWWUE6H3OQSGAQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.344916562 (383h23m57.215999597s) [noise=true , address=77.90.31.12:4590 , piece_id=ISJYEFBJZXQYMTRVNGM5UGHUWJKWRKXHBKLJ4WOEU2HEGXJV5GFQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.345654427 (383h23m57.216737462s) [noise=true , address=145.79.227.50:4375 , piece_id=ZP4M6EQCM4BVBTWMZXEKEQ7CTVJQ2VDY53T4CCJNP7Q4ERJGXRHA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.346243378 (383h23m57.217326423s) [noise=true , address=145.79.227.37:4565 , piece_id=7ORBYHMU6TPTHUX63O53XS5JDK5CRC3DTM7QTULG6TLEHUALHOTA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.346376732 (383h23m57.217459777s) [noise=true , address=145.79.227.27:4690 , piece_id=FFMPXFCKUEFZY7OJWA7ZNJYLULCY6ZELNYW2L7BLDVUNIN723DQA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.346693024 (383h23m57.21777606s) [noise=true , address=145.79.227.52:4445 , piece_id=FYQ24RQLYTOW3633HO4OKX5IT5HVNFELT4IWQN3YB26VDSOVXUYQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.346822732 (383h23m57.217905777s) [noise=true , address=45.91.116.8:4045 , piece_id=65MKE37HLBKEVQBETT3QG4XITTRAMIBMMZM4NZC5JAOJM7LISLYQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.347134075 (383h23m57.21821711s) [noise=true , address=145.79.227.14:4838 , piece_id=AOCWTGDXJUOQYLPCZIHM2TCAPRHPK7HXRT3SZTJHN5M7SCB3LG6Q, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.347258822 (383h23m57.218341867s) [noise=true , address=145.79.227.34:4907 , piece_id=S2VQJ3M3HO7EWSFMWTVT5VGTTUUK4ZHP6ID2Z6KCY4DVYSJNBNFA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.347696295 (383h23m57.21877933s) [noise=true , address=145.79.227.9:4366 , piece_id=X7G5XC2TCIN6UC5UOGPTOZCT3ZGKLJA7Y6RQTBYNHRK6YY2BZ2JA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.347731111 (383h23m57.218814146s) [noise=true , address=145.79.227.42:4121 , piece_id=2UDMPQ3XM3P7NISLAA74YVAZMRQUKYOULMJHMP5YHDRNJ5WDUYWQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.348159457 (383h23m57.219242502s) [noise=true , address=145.79.227.40:4051 , piece_id=6PVPS3SN4MYYMI33CQCFRNYAZXSJMATQWXJ67TUD6TAOBFR2LJVA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.348210303 (383h23m57.219293338s) [noise=true , address=77.90.31.11:4553 , piece_id=3LDMHDTERLAWB6ZQGO3GHCJJCOJRT76JWBJGHW4YHKU2L55ZKAEQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.348734782 (383h23m57.219817817s) [noise=true , address=145.79.227.35:4481 , piece_id=CNAMJ4HUKBMCEWKIOAEGAF5ZX7IWT5PYMY5NRLNAZUVGS25UVERA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.349213964 (383h23m57.220296999s) [noise=true , address=145.79.227.46:4603 , piece_id=LCP2YLCB6NZEQHP6UNQTHQ4GHDM4FT7SVWBRHISDIQFR6QLW3A7A, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.349717554 (383h23m57.220800599s) [noise=true , address=145.79.227.51:4411 , piece_id=GNWN2W5VCM6AE24AQK2WNSE6DRITBXE2Y2BO2JVSAQZOQIBGO7MA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:59.214236472 (383h23m59.085319527s) [upload=336495]: Next returned nil segment Dec 21 14:52:59.214252453 (383h23m59.085335498s) [upload=336495]: Waiting for error group managing segments... Dec 21 14:52:59.555201717 (383h23m59.426284752s) [noise=true , address=145.79.227.52:4445 , piece_id=FYQ24RQLYTOW3633HO4OKX5IT5HVNFELT4IWQN3YB26VDSOVXUYQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:52:59.556541508 (383h23m59.427624553s) [noise=true , address=145.79.227.50:4375 , piece_id=ZP4M6EQCM4BVBTWMZXEKEQ7CTVJQ2VDY53T4CCJNP7Q4ERJGXRHA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:52:59.599023710 (383h23m59.470106745s) [noise=true , address=77.90.31.12:4590 , piece_id=ISJYEFBJZXQYMTRVNGM5UGHUWJKWRKXHBKLJ4WOEU2HEGXJV5GFQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:52:59.607965507 (383h23m59.479048552s) [noise=true , address=77.90.31.11:4553 , piece_id=3LDMHDTERLAWB6ZQGO3GHCJJCOJRT76JWBJGHW4YHKU2L55ZKAEQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:00.713826993 (383h24m0.584910028s) [noise=true , address=145.79.227.46:4603 , piece_id=LCP2YLCB6NZEQHP6UNQTHQ4GHDM4FT7SVWBRHISDIQFR6QLW3A7A, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:00.956673469 (383h24m0.827756514s) [noise=true , address=145.79.227.34:4907 , piece_id=S2VQJ3M3HO7EWSFMWTVT5VGTTUUK4ZHP6ID2Z6KCY4DVYSJNBNFA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:01.357775103 (383h24m1.228858148s) [noise=true , address=45.91.116.8:4045 , piece_id=65MKE37HLBKEVQBETT3QG4XITTRAMIBMMZM4NZC5JAOJM7LISLYQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:01.654871136 (383h24m1.525954171s) [noise=true , address=145.79.227.9:4366 , piece_id=X7G5XC2TCIN6UC5UOGPTOZCT3ZGKLJA7Y6RQTBYNHRK6YY2BZ2JA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:01.813911231 (383h24m1.684994276s) [noise=true , address=45.91.116.7:4009 , piece_id=XR4FURKTH7PLUFFFT4ZK4RFPNUWMR6O7FKMLDURWWUE6H3OQSGAQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:01.977475780 (383h24m1.848558835s) [noise=true , address=145.79.227.42:4121 , piece_id=2UDMPQ3XM3P7NISLAA74YVAZMRQUKYOULMJHMP5YHDRNJ5WDUYWQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:02.443864100 (383h24m2.314947155s) [noise=true , address=145.79.227.37:4565 , piece_id=7ORBYHMU6TPTHUX63O53XS5JDK5CRC3DTM7QTULG6TLEHUALHOTA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:02.664185743 (383h24m2.535268788s) [noise=true , address=145.79.227.14:4838 , piece_id=AOCWTGDXJUOQYLPCZIHM2TCAPRHPK7HXRT3SZTJHN5M7SCB3LG6Q, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:03.154784372 (383h24m3.025867417s) [noise=true , address=145.79.227.51:4411 , piece_id=GNWN2W5VCM6AE24AQK2WNSE6DRITBXE2Y2BO2JVSAQZOQIBGO7MA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:03.154806574 (383h24m3.025889609s) [seg_pos={0 0}, upload=336495]: Segment reached optimal threshold of 13 pieces. Dec 21 14:53:03.154980986 (383h24m3.026064022s) [noise=true , address=145.79.227.40:4051 , piece_id=6PVPS3SN4MYYMI33CQCFRNYAZXSJMATQWXJ67TUD6TAOBFR2LJVA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: write tcp 188.166.225.170:47278->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:47278->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47278->145.79.227.40:4051: use of closed network connection Dec 21 14:53:03.154987609 (383h24m3.026070644s) [noise=true , address=145.79.227.40:4051 , piece_id=6PVPS3SN4MYYMI33CQCFRNYAZXSJMATQWXJ67TUD6TAOBFR2LJVA, seg_pos={0 0}, upload=336495]: Long tail cancellation due to optimal threshold Dec 21 14:53:03.154998870 (383h24m3.026081915s) [noise=true , address=145.79.227.35:4481 , piece_id=CNAMJ4HUKBMCEWKIOAEGAF5ZX7IWT5PYMY5NRLNAZUVGS25UVERA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:43498->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:43498->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43498->145.79.227.35:4481: use of closed network connection Dec 21 14:53:03.155010201 (383h24m3.026093246s) [noise=true , address=145.79.227.35:4481 , piece_id=CNAMJ4HUKBMCEWKIOAEGAF5ZX7IWT5PYMY5NRLNAZUVGS25UVERA, seg_pos={0 0}, upload=336495]: Long tail cancellation due to optimal threshold Dec 21 14:53:03.155078662 (383h24m3.026161707s) [noise=true , address=145.79.227.27:4690 , piece_id=FFMPXFCKUEFZY7OJWA7ZNJYLULCY6ZELNYW2L7BLDVUNIN723DQA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:34208->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:34208->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34208->145.79.227.27:4690: use of closed network connection Dec 21 14:53:03.155084243 (383h24m3.026167288s) [noise=true , address=145.79.227.27:4690 , piece_id=FFMPXFCKUEFZY7OJWA7ZNJYLULCY6ZELNYW2L7BLDVUNIN723DQA, seg_pos={0 0}, upload=336495]: Long tail cancellation due to optimal threshold Dec 21 14:53:03.155101896 (383h24m3.026184941s) [seg_pos={0 0}, upload=336495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:53:03.155184433 (383h24m3.026267468s) [upload=336495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:53:03.155189684 (383h24m3.026272729s) [upload=336495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:53:03.807547284 (383h24m3.678630329s) [upload=336495]: Done uploading segments... Dec 21 14:53:03.807572753 (383h24m3.678655798s) [upload=336495]: Upload finished. err: <nil> Duration:* 5.869139447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X..X.....X......X...................X.........X....X] duration: 30.002528933s trace: https://jaeger.dev.storj.io/trace/4c3c6311ccf71a70 FAIL (2025-12-21 14:53:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: 2UDMPQ3XM3P7NISLAA74YVAZMRQUKYOULMJHMP5YHDRNJ5WDUYWQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: LCP2YLCB6NZEQHP6UNQTHQ4GHDM4FT7SVWBRHISDIQFR6QLW3A7A) context deadline exceeded; piecestore: (Node ID: 1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, Piece ID: GNWN2W5VCM6AE24AQK2WNSE6DRITBXE2Y2BO2JVSAQZOQIBGO7MA) context deadline exceeded ======LOGS====== 57.040323002 (383h23m56.911406047s) [upload=336495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 14:52:57.344790141 (383h23m57.215873186s) [seg_pos={0 0}, upload=336495]: Begin upload segment... Dec 21 14:52:57.344861838 (383h23m57.215967947s) [seg_pos={0 0}, upload=336495]: Done begin upload segment. Dec 21 14:52:57.344894730 (383h23m57.215977765s) [noise=true , address=45.91.116.7:4009 , piece_id=XR4FURKTH7PLUFFFT4ZK4RFPNUWMR6O7FKMLDURWWUE6H3OQSGAQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.344916562 (383h23m57.215999597s) [noise=true , address=77.90.31.12:4590 , piece_id=ISJYEFBJZXQYMTRVNGM5UGHUWJKWRKXHBKLJ4WOEU2HEGXJV5GFQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.345654427 (383h23m57.216737462s) [noise=true , address=145.79.227.50:4375 , piece_id=ZP4M6EQCM4BVBTWMZXEKEQ7CTVJQ2VDY53T4CCJNP7Q4ERJGXRHA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.346243378 (383h23m57.217326423s) [noise=true , address=145.79.227.37:4565 , piece_id=7ORBYHMU6TPTHUX63O53XS5JDK5CRC3DTM7QTULG6TLEHUALHOTA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.346376732 (383h23m57.217459777s) [noise=true , address=145.79.227.27:4690 , piece_id=FFMPXFCKUEFZY7OJWA7ZNJYLULCY6ZELNYW2L7BLDVUNIN723DQA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.346693024 (383h23m57.21777606s) [noise=true , address=145.79.227.52:4445 , piece_id=FYQ24RQLYTOW3633HO4OKX5IT5HVNFELT4IWQN3YB26VDSOVXUYQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.346822732 (383h23m57.217905777s) [noise=true , address=45.91.116.8:4045 , piece_id=65MKE37HLBKEVQBETT3QG4XITTRAMIBMMZM4NZC5JAOJM7LISLYQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.347134075 (383h23m57.21821711s) [noise=true , address=145.79.227.14:4838 , piece_id=AOCWTGDXJUOQYLPCZIHM2TCAPRHPK7HXRT3SZTJHN5M7SCB3LG6Q, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.347258822 (383h23m57.218341867s) [noise=true , address=145.79.227.34:4907 , piece_id=S2VQJ3M3HO7EWSFMWTVT5VGTTUUK4ZHP6ID2Z6KCY4DVYSJNBNFA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.347696295 (383h23m57.21877933s) [noise=true , address=145.79.227.9:4366 , piece_id=X7G5XC2TCIN6UC5UOGPTOZCT3ZGKLJA7Y6RQTBYNHRK6YY2BZ2JA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.347731111 (383h23m57.218814146s) [noise=true , address=145.79.227.42:4121 , piece_id=2UDMPQ3XM3P7NISLAA74YVAZMRQUKYOULMJHMP5YHDRNJ5WDUYWQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.348159457 (383h23m57.219242502s) [noise=true , address=145.79.227.40:4051 , piece_id=6PVPS3SN4MYYMI33CQCFRNYAZXSJMATQWXJ67TUD6TAOBFR2LJVA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.348210303 (383h23m57.219293338s) [noise=true , address=77.90.31.11:4553 , piece_id=3LDMHDTERLAWB6ZQGO3GHCJJCOJRT76JWBJGHW4YHKU2L55ZKAEQ, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.348734782 (383h23m57.219817817s) [noise=true , address=145.79.227.35:4481 , piece_id=CNAMJ4HUKBMCEWKIOAEGAF5ZX7IWT5PYMY5NRLNAZUVGS25UVERA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.349213964 (383h23m57.220296999s) [noise=true , address=145.79.227.46:4603 , piece_id=LCP2YLCB6NZEQHP6UNQTHQ4GHDM4FT7SVWBRHISDIQFR6QLW3A7A, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:57.349717554 (383h23m57.220800599s) [noise=true , address=145.79.227.51:4411 , piece_id=GNWN2W5VCM6AE24AQK2WNSE6DRITBXE2Y2BO2JVSAQZOQIBGO7MA, seg_pos={0 0}, upload=336495]: Uploading piece... Dec 21 14:52:59.214236472 (383h23m59.085319527s) [upload=336495]: Next returned nil segment Dec 21 14:52:59.214252453 (383h23m59.085335498s) [upload=336495]: Waiting for error group managing segments... Dec 21 14:52:59.555201717 (383h23m59.426284752s) [noise=true , address=145.79.227.52:4445 , piece_id=FYQ24RQLYTOW3633HO4OKX5IT5HVNFELT4IWQN3YB26VDSOVXUYQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:52:59.556541508 (383h23m59.427624553s) [noise=true , address=145.79.227.50:4375 , piece_id=ZP4M6EQCM4BVBTWMZXEKEQ7CTVJQ2VDY53T4CCJNP7Q4ERJGXRHA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:52:59.599023710 (383h23m59.470106745s) [noise=true , address=77.90.31.12:4590 , piece_id=ISJYEFBJZXQYMTRVNGM5UGHUWJKWRKXHBKLJ4WOEU2HEGXJV5GFQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:52:59.607965507 (383h23m59.479048552s) [noise=true , address=77.90.31.11:4553 , piece_id=3LDMHDTERLAWB6ZQGO3GHCJJCOJRT76JWBJGHW4YHKU2L55ZKAEQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:00.713826993 (383h24m0.584910028s) [noise=true , address=145.79.227.46:4603 , piece_id=LCP2YLCB6NZEQHP6UNQTHQ4GHDM4FT7SVWBRHISDIQFR6QLW3A7A, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:00.956673469 (383h24m0.827756514s) [noise=true , address=145.79.227.34:4907 , piece_id=S2VQJ3M3HO7EWSFMWTVT5VGTTUUK4ZHP6ID2Z6KCY4DVYSJNBNFA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:01.357775103 (383h24m1.228858148s) [noise=true , address=45.91.116.8:4045 , piece_id=65MKE37HLBKEVQBETT3QG4XITTRAMIBMMZM4NZC5JAOJM7LISLYQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:01.654871136 (383h24m1.525954171s) [noise=true , address=145.79.227.9:4366 , piece_id=X7G5XC2TCIN6UC5UOGPTOZCT3ZGKLJA7Y6RQTBYNHRK6YY2BZ2JA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:01.813911231 (383h24m1.684994276s) [noise=true , address=45.91.116.7:4009 , piece_id=XR4FURKTH7PLUFFFT4ZK4RFPNUWMR6O7FKMLDURWWUE6H3OQSGAQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:01.977475780 (383h24m1.848558835s) [noise=true , address=145.79.227.42:4121 , piece_id=2UDMPQ3XM3P7NISLAA74YVAZMRQUKYOULMJHMP5YHDRNJ5WDUYWQ, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:02.443864100 (383h24m2.314947155s) [noise=true , address=145.79.227.37:4565 , piece_id=7ORBYHMU6TPTHUX63O53XS5JDK5CRC3DTM7QTULG6TLEHUALHOTA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:02.664185743 (383h24m2.535268788s) [noise=true , address=145.79.227.14:4838 , piece_id=AOCWTGDXJUOQYLPCZIHM2TCAPRHPK7HXRT3SZTJHN5M7SCB3LG6Q, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:03.154784372 (383h24m3.025867417s) [noise=true , address=145.79.227.51:4411 , piece_id=GNWN2W5VCM6AE24AQK2WNSE6DRITBXE2Y2BO2JVSAQZOQIBGO7MA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: <nil> Dec 21 14:53:03.154806574 (383h24m3.025889609s) [seg_pos={0 0}, upload=336495]: Segment reached optimal threshold of 13 pieces. Dec 21 14:53:03.154980986 (383h24m3.026064022s) [noise=true , address=145.79.227.40:4051 , piece_id=6PVPS3SN4MYYMI33CQCFRNYAZXSJMATQWXJ67TUD6TAOBFR2LJVA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: write tcp 188.166.225.170:47278->145.79.227.40:4051: use of closed network connection; write tcp 188.166.225.170:47278->145.79.227.40:4051: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47278->145.79.227.40:4051: use of closed network connection Dec 21 14:53:03.154987609 (383h24m3.026070644s) [noise=true , address=145.79.227.40:4051 , piece_id=6PVPS3SN4MYYMI33CQCFRNYAZXSJMATQWXJ67TUD6TAOBFR2LJVA, seg_pos={0 0}, upload=336495]: Long tail cancellation due to optimal threshold Dec 21 14:53:03.154998870 (383h24m3.026081915s) [noise=true , address=145.79.227.35:4481 , piece_id=CNAMJ4HUKBMCEWKIOAEGAF5ZX7IWT5PYMY5NRLNAZUVGS25UVERA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:43498->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:43498->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43498->145.79.227.35:4481: use of closed network connection Dec 21 14:53:03.155010201 (383h24m3.026093246s) [noise=true , address=145.79.227.35:4481 , piece_id=CNAMJ4HUKBMCEWKIOAEGAF5ZX7IWT5PYMY5NRLNAZUVGS25UVERA, seg_pos={0 0}, upload=336495]: Long tail cancellation due to optimal threshold Dec 21 14:53:03.155078662 (383h24m3.026161707s) [noise=true , address=145.79.227.27:4690 , piece_id=FFMPXFCKUEFZY7OJWA7ZNJYLULCY6ZELNYW2L7BLDVUNIN723DQA, seg_pos={0 0}, upload=336495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:34208->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:34208->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34208->145.79.227.27:4690: use of closed network connection Dec 21 14:53:03.155084243 (383h24m3.026167288s) [noise=true , address=145.79.227.27:4690 , piece_id=FFMPXFCKUEFZY7OJWA7ZNJYLULCY6ZELNYW2L7BLDVUNIN723DQA, seg_pos={0 0}, upload=336495]: Long tail cancellation due to optimal threshold Dec 21 14:53:03.155101896 (383h24m3.026184941s) [seg_pos={0 0}, upload=336495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 14:53:03.155184433 (383h24m3.026267468s) [upload=336495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:53:03.155189684 (383h24m3.026272729s) [upload=336495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:53:03.807547284 (383h24m3.678630329s) [upload=336495]: Done uploading segments... Dec 21 14:53:03.807572753 (383h24m3.678655798s) [upload=336495]: Upload finished. err: <nil> Duration:* 30.00255323s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)