Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration: 35.610954343s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:16:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 35.610972738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.117800256s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:16:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.117856022s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X.........X....X.X......XX............X..XX....] duration: 26.511905689s trace: https://jaeger.dev.storj.io/trace/555b98afc45f369b FAIL (2025-12-21 15:16:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: 4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q) context deadline exceeded ======LOGS====== 21 15:12:54.007507162 (383h43m53.878590207s) [upload=336789]: Starting upload Dec 21 15:12:54.007585932 (383h43m53.878668967s) [upload=336789]: Done starting upload Dec 21 15:12:54.007679150 (383h43m53.878762195s) [upload=336789]: Uploading segments... Dec 21 15:12:54.007819236 (383h43m53.878902271s) [upload=336789]: Got next segment. Inline: false Dec 21 15:12:54.007870023 (383h43m53.878953068s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:12:54.007875904 (383h43m53.878958939s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:12:54.258193935 (383h43m54.12927697s) [seg_pos={0 0}, upload=336789]: Begin upload segment... Dec 21 15:12:54.258246765 (383h43m54.12932981s) [seg_pos={0 0}, upload=336789]: Done begin upload segment. Dec 21 15:12:54.258275840 (383h43m54.129358875s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258350753 (383h43m54.129433798s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258811751 (383h43m54.129894786s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259112263 (383h43m54.130195298s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259409950 (383h43m54.130492995s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259692147 (383h43m54.130775192s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259983953 (383h43m54.131066998s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260324992 (383h43m54.131408027s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260659680 (383h43m54.131742725s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.262299803 (383h43m54.133382848s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.263642740 (383h43m54.134725775s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.264654726 (383h43m54.135737762s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.265930837 (383h43m54.137013882s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.267100815 (383h43m54.13818385s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.268237308 (383h43m54.139320343s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.269391347 (383h43m54.140474392s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:56.113823557 (383h43m55.984906592s) [upload=336789]: Next returned nil segment Dec 21 15:12:56.113837995 (383h43m55.98492103s) [upload=336789]: Waiting for error group managing segments... Dec 21 15:12:56.416591731 (383h43m56.287674766s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.421193173 (383h43m56.292276208s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.424155363 (383h43m56.295238398s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.501472225 (383h43m56.37255527s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.513791723 (383h43m56.384874768s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.523529074 (383h43m56.394612109s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.530067665 (383h43m56.4011507s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.728286271 (383h43m56.599369306s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.823163831 (383h43m56.694246866s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.882961134 (383h43m56.754044179s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.114415925 (383h43m56.985498961s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.171299290 (383h43m57.042382325s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460538793 (383h43m57.331621828s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460561586 (383h43m57.331644621s) [seg_pos={0 0}, upload=336789]: Segment reached optimal threshold of 13 pieces. Dec 21 15:12:57.460799409 (383h43m57.331882474s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460810501 (383h43m57.331893536s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.460896294 (383h43m57.331979339s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460916893 (383h43m57.331999958s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461092117 (383h43m57.332175172s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection Dec 21 15:12:57.461101645 (383h43m57.3321847s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461120069 (383h43m57.332203124s) [seg_pos={0 0}, upload=336789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:12:57.461220771 (383h43m57.332303816s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:12:57.461227725 (383h43m57.33231077s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:12:57.759106067 (383h43m57.630189122s) [upload=336789]: Done uploading segments... Dec 21 15:12:57.759131094 (383h43m57.630214139s) [upload=336789]: Upload finished. err: <nil> Duration: 26.511930336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.X.................................................] duration: 33.336316598s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:15:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 33.336346695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............X.........X....X.X......XX............X..XX...] duration: 15.850213329s trace: https://jaeger.dev.storj.io/trace/555b98afc45f369b FAIL (2025-12-21 15:15:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: 4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q) context deadline exceeded ======LOGS====== 21 15:12:54.007507162 (383h43m53.878590207s) [upload=336789]: Starting upload Dec 21 15:12:54.007585932 (383h43m53.878668967s) [upload=336789]: Done starting upload Dec 21 15:12:54.007679150 (383h43m53.878762195s) [upload=336789]: Uploading segments... Dec 21 15:12:54.007819236 (383h43m53.878902271s) [upload=336789]: Got next segment. Inline: false Dec 21 15:12:54.007870023 (383h43m53.878953068s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:12:54.007875904 (383h43m53.878958939s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:12:54.258193935 (383h43m54.12927697s) [seg_pos={0 0}, upload=336789]: Begin upload segment... Dec 21 15:12:54.258246765 (383h43m54.12932981s) [seg_pos={0 0}, upload=336789]: Done begin upload segment. Dec 21 15:12:54.258275840 (383h43m54.129358875s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258350753 (383h43m54.129433798s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258811751 (383h43m54.129894786s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259112263 (383h43m54.130195298s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259409950 (383h43m54.130492995s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259692147 (383h43m54.130775192s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259983953 (383h43m54.131066998s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260324992 (383h43m54.131408027s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260659680 (383h43m54.131742725s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.262299803 (383h43m54.133382848s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.263642740 (383h43m54.134725775s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.264654726 (383h43m54.135737762s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.265930837 (383h43m54.137013882s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.267100815 (383h43m54.13818385s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.268237308 (383h43m54.139320343s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.269391347 (383h43m54.140474392s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:56.113823557 (383h43m55.984906592s) [upload=336789]: Next returned nil segment Dec 21 15:12:56.113837995 (383h43m55.98492103s) [upload=336789]: Waiting for error group managing segments... Dec 21 15:12:56.416591731 (383h43m56.287674766s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.421193173 (383h43m56.292276208s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.424155363 (383h43m56.295238398s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.501472225 (383h43m56.37255527s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.513791723 (383h43m56.384874768s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.523529074 (383h43m56.394612109s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.530067665 (383h43m56.4011507s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.728286271 (383h43m56.599369306s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.823163831 (383h43m56.694246866s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.882961134 (383h43m56.754044179s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.114415925 (383h43m56.985498961s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.171299290 (383h43m57.042382325s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460538793 (383h43m57.331621828s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460561586 (383h43m57.331644621s) [seg_pos={0 0}, upload=336789]: Segment reached optimal threshold of 13 pieces. Dec 21 15:12:57.460799409 (383h43m57.331882474s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460810501 (383h43m57.331893536s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.460896294 (383h43m57.331979339s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460916893 (383h43m57.331999958s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461092117 (383h43m57.332175172s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection Dec 21 15:12:57.461101645 (383h43m57.3321847s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461120069 (383h43m57.332203124s) [seg_pos={0 0}, upload=336789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:12:57.461220771 (383h43m57.332303816s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:12:57.461227725 (383h43m57.33231077s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:12:57.759106067 (383h43m57.630189122s) [upload=336789]: Done uploading segments... Dec 21 15:12:57.759131094 (383h43m57.630214139s) [upload=336789]: Upload finished. err: <nil> Duration: 15.850248036s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.212486705s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:15:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.212514821s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.X................................................] duration: 28.377630679s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:15:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 28.377646539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.X...............................................] duration: 33.748204903s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:14:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 33.748226704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.120353012s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:14:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.120406464s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.X..............................................] duration: 33.354512334s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:14:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 33.354541199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............X.........X....X.X......XX............X..XX..] duration: 5.877170651s trace: https://jaeger.dev.storj.io/trace/555b98afc45f369b FAIL (2025-12-21 15:13:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: 4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q) context deadline exceeded ======LOGS====== 21 15:12:54.007507162 (383h43m53.878590207s) [upload=336789]: Starting upload Dec 21 15:12:54.007585932 (383h43m53.878668967s) [upload=336789]: Done starting upload Dec 21 15:12:54.007679150 (383h43m53.878762195s) [upload=336789]: Uploading segments... Dec 21 15:12:54.007819236 (383h43m53.878902271s) [upload=336789]: Got next segment. Inline: false Dec 21 15:12:54.007870023 (383h43m53.878953068s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:12:54.007875904 (383h43m53.878958939s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:12:54.258193935 (383h43m54.12927697s) [seg_pos={0 0}, upload=336789]: Begin upload segment... Dec 21 15:12:54.258246765 (383h43m54.12932981s) [seg_pos={0 0}, upload=336789]: Done begin upload segment. Dec 21 15:12:54.258275840 (383h43m54.129358875s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258350753 (383h43m54.129433798s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258811751 (383h43m54.129894786s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259112263 (383h43m54.130195298s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259409950 (383h43m54.130492995s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259692147 (383h43m54.130775192s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259983953 (383h43m54.131066998s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260324992 (383h43m54.131408027s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260659680 (383h43m54.131742725s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.262299803 (383h43m54.133382848s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.263642740 (383h43m54.134725775s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.264654726 (383h43m54.135737762s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.265930837 (383h43m54.137013882s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.267100815 (383h43m54.13818385s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.268237308 (383h43m54.139320343s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.269391347 (383h43m54.140474392s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:56.113823557 (383h43m55.984906592s) [upload=336789]: Next returned nil segment Dec 21 15:12:56.113837995 (383h43m55.98492103s) [upload=336789]: Waiting for error group managing segments... Dec 21 15:12:56.416591731 (383h43m56.287674766s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.421193173 (383h43m56.292276208s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.424155363 (383h43m56.295238398s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.501472225 (383h43m56.37255527s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.513791723 (383h43m56.384874768s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.523529074 (383h43m56.394612109s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.530067665 (383h43m56.4011507s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.728286271 (383h43m56.599369306s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.823163831 (383h43m56.694246866s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.882961134 (383h43m56.754044179s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.114415925 (383h43m56.985498961s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.171299290 (383h43m57.042382325s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460538793 (383h43m57.331621828s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460561586 (383h43m57.331644621s) [seg_pos={0 0}, upload=336789]: Segment reached optimal threshold of 13 pieces. Dec 21 15:12:57.460799409 (383h43m57.331882474s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460810501 (383h43m57.331893536s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.460896294 (383h43m57.331979339s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460916893 (383h43m57.331999958s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461092117 (383h43m57.332175172s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection Dec 21 15:12:57.461101645 (383h43m57.3321847s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461120069 (383h43m57.332203124s) [seg_pos={0 0}, upload=336789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:12:57.461220771 (383h43m57.332303816s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:12:57.461227725 (383h43m57.33231077s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:12:57.759106067 (383h43m57.630189122s) [upload=336789]: Done uploading segments... Dec 21 15:12:57.759131094 (383h43m57.630214139s) [upload=336789]: Upload finished. err: <nil> Duration: 5.877190177s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [................X.........X....X.X......XX............X..XX.] duration: 22.44174848s trace: https://jaeger.dev.storj.io/trace/555b98afc45f369b FAIL (2025-12-21 15:13:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: 4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q) context deadline exceeded ======LOGS====== 21 15:12:54.007507162 (383h43m53.878590207s) [upload=336789]: Starting upload Dec 21 15:12:54.007585932 (383h43m53.878668967s) [upload=336789]: Done starting upload Dec 21 15:12:54.007679150 (383h43m53.878762195s) [upload=336789]: Uploading segments... Dec 21 15:12:54.007819236 (383h43m53.878902271s) [upload=336789]: Got next segment. Inline: false Dec 21 15:12:54.007870023 (383h43m53.878953068s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:12:54.007875904 (383h43m53.878958939s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:12:54.258193935 (383h43m54.12927697s) [seg_pos={0 0}, upload=336789]: Begin upload segment... Dec 21 15:12:54.258246765 (383h43m54.12932981s) [seg_pos={0 0}, upload=336789]: Done begin upload segment. Dec 21 15:12:54.258275840 (383h43m54.129358875s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258350753 (383h43m54.129433798s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258811751 (383h43m54.129894786s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259112263 (383h43m54.130195298s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259409950 (383h43m54.130492995s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259692147 (383h43m54.130775192s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259983953 (383h43m54.131066998s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260324992 (383h43m54.131408027s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260659680 (383h43m54.131742725s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.262299803 (383h43m54.133382848s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.263642740 (383h43m54.134725775s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.264654726 (383h43m54.135737762s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.265930837 (383h43m54.137013882s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.267100815 (383h43m54.13818385s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.268237308 (383h43m54.139320343s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.269391347 (383h43m54.140474392s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:56.113823557 (383h43m55.984906592s) [upload=336789]: Next returned nil segment Dec 21 15:12:56.113837995 (383h43m55.98492103s) [upload=336789]: Waiting for error group managing segments... Dec 21 15:12:56.416591731 (383h43m56.287674766s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.421193173 (383h43m56.292276208s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.424155363 (383h43m56.295238398s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.501472225 (383h43m56.37255527s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.513791723 (383h43m56.384874768s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.523529074 (383h43m56.394612109s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.530067665 (383h43m56.4011507s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.728286271 (383h43m56.599369306s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.823163831 (383h43m56.694246866s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.882961134 (383h43m56.754044179s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.114415925 (383h43m56.985498961s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.171299290 (383h43m57.042382325s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460538793 (383h43m57.331621828s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460561586 (383h43m57.331644621s) [seg_pos={0 0}, upload=336789]: Segment reached optimal threshold of 13 pieces. Dec 21 15:12:57.460799409 (383h43m57.331882474s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460810501 (383h43m57.331893536s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.460896294 (383h43m57.331979339s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460916893 (383h43m57.331999958s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461092117 (383h43m57.332175172s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection Dec 21 15:12:57.461101645 (383h43m57.3321847s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461120069 (383h43m57.332203124s) [seg_pos={0 0}, upload=336789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:12:57.461220771 (383h43m57.332303816s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:12:57.461227725 (383h43m57.33231077s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:12:57.759106067 (383h43m57.630189122s) [upload=336789]: Done uploading segments... Dec 21 15:12:57.759131094 (383h43m57.630214139s) [upload=336789]: Upload finished. err: <nil> Duration: 22.441781362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.................X.........X....X.X......XX............X..XX] duration: 30.003419437s trace: https://jaeger.dev.storj.io/trace/555b98afc45f369b FAIL (2025-12-21 15:13:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ) context deadline exceeded; piecestore: (Node ID: 12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, Piece ID: 4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q) context deadline exceeded ======LOGS====== 21 15:12:54.007507162 (383h43m53.878590207s) [upload=336789]: Starting upload Dec 21 15:12:54.007585932 (383h43m53.878668967s) [upload=336789]: Done starting upload Dec 21 15:12:54.007679150 (383h43m53.878762195s) [upload=336789]: Uploading segments... Dec 21 15:12:54.007819236 (383h43m53.878902271s) [upload=336789]: Got next segment. Inline: false Dec 21 15:12:54.007870023 (383h43m53.878953068s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:12:54.007875904 (383h43m53.878958939s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:12:54.258193935 (383h43m54.12927697s) [seg_pos={0 0}, upload=336789]: Begin upload segment... Dec 21 15:12:54.258246765 (383h43m54.12932981s) [seg_pos={0 0}, upload=336789]: Done begin upload segment. Dec 21 15:12:54.258275840 (383h43m54.129358875s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258350753 (383h43m54.129433798s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.258811751 (383h43m54.129894786s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259112263 (383h43m54.130195298s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259409950 (383h43m54.130492995s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259692147 (383h43m54.130775192s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.259983953 (383h43m54.131066998s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260324992 (383h43m54.131408027s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.260659680 (383h43m54.131742725s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.262299803 (383h43m54.133382848s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.263642740 (383h43m54.134725775s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.264654726 (383h43m54.135737762s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.265930837 (383h43m54.137013882s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.267100815 (383h43m54.13818385s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.268237308 (383h43m54.139320343s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:54.269391347 (383h43m54.140474392s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Uploading piece... Dec 21 15:12:56.113823557 (383h43m55.984906592s) [upload=336789]: Next returned nil segment Dec 21 15:12:56.113837995 (383h43m55.98492103s) [upload=336789]: Waiting for error group managing segments... Dec 21 15:12:56.416591731 (383h43m56.287674766s) [noise=true , address=145.79.227.42:4123 , piece_id=O4SJIKBZEUPGFREDXGJLRLM2PRSKWDC5U7JGOGUXI7UT6GGYSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.421193173 (383h43m56.292276208s) [noise=true , address=145.79.227.46:4603 , piece_id=QVPBEQ77T575I2RULPDGTB3XZE5TWGOVQT4LJILGCKY35XUMI6XA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.424155363 (383h43m56.295238398s) [noise=true , address=145.79.227.38:4978 , piece_id=5H4UTXCYRUCH6OWXOK2VEOPUGIF5CDFUEV7BL57GP4TFGU57LBRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.501472225 (383h43m56.37255527s) [noise=true , address=145.79.227.35:4482 , piece_id=OVYWWQD227DQL22ZJQG2BMGAN4AIV53BB2HGFKTANNZA4HWD2C2Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.513791723 (383h43m56.384874768s) [noise=true , address=145.79.227.52:4447 , piece_id=SLGFLLD7SFBY3BSYINC47P4XHTYUT4JCU4NHRBH7QFRODI4SIB3A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.523529074 (383h43m56.394612109s) [noise=true , address=45.91.116.5:4299 , piece_id=QRATNEUWSNDY3IDXJYB6XFXTDMBNVSKRBC3YJDSHJFD2ZJ7QS7WQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.530067665 (383h43m56.4011507s) [noise=true , address=45.91.116.10:4081 , piece_id=WVSDSYSUK4IIP5ZYRHZMUZATCLOM3G3F5G6UZR3RU7ES7OCJSVQQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.728286271 (383h43m56.599369306s) [noise=true , address=145.79.227.51:4409 , piece_id=FD6SJI5MLABVVTT5G55VXWZ6JSFFWVDWWMNH2D7J7KZ5IV3L3X5Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.823163831 (383h43m56.694246866s) [noise=true , address=145.79.227.47:4267 , piece_id=ZDXLESVID2Q22I2QDZFLDFUPIYKEOEPSUJCSTMCLNDL2UJSHGMRQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:56.882961134 (383h43m56.754044179s) [noise=true , address=45.91.116.14:4192 , piece_id=QOLZ5XVGAVX5A3XOCC2ALYKRGYXPSFCQOC7WTFZYR2WVLNZTQUSA, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.114415925 (383h43m56.985498961s) [noise=true , address=145.79.227.27:4690 , piece_id=U2HUBWBIIBFOZGIRPH627JU3T2YD4OP5KTPCMIPCUUMMYWDTHQNQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.171299290 (383h43m57.042382325s) [noise=true , address=145.79.227.25:4617 , piece_id=4RJIE6EMKWE476KO2UE3HB2LCGTW6PNPGCJD6IOI3PQLF2CGD34Q, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460538793 (383h43m57.331621828s) [noise=true , address=145.79.227.14:4838 , piece_id=2XTDJQSJMIYAV6CHXUCVVDOYQQGAZWKKY4QL564SEOFOH2CX7FKQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: <nil> Dec 21 15:12:57.460561586 (383h43m57.331644621s) [seg_pos={0 0}, upload=336789]: Segment reached optimal threshold of 13 pieces. Dec 21 15:12:57.460799409 (383h43m57.331882474s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460810501 (383h43m57.331893536s) [noise=true , address=145.79.227.39:4015 , piece_id=YFFQKVEU7EDDIOYZQUYLIPD7TFPIO7ZLYSW4ZAI6FRMAZ5EP2IFQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.460896294 (383h43m57.331979339s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:57.460916893 (383h43m57.331999958s) [noise=true , address=145.79.227.44:4194 , piece_id=CHWJMQ76Q24SQXOCJWUIJ3BXHNHC3HVNIVIW2WDDWYCEEDLJZW4A, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461092117 (383h43m57.332175172s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45830->145.79.227.28:4727: use of closed network connection Dec 21 15:12:57.461101645 (383h43m57.3321847s) [noise=true , address=145.79.227.28:4727 , piece_id=3BLGCRHAUXPJR67CC3ZSNCUK75NJS2OXMXU6TQBSMXA7FQBWX3IQ, seg_pos={0 0}, upload=336789]: Long tail cancellation due to optimal threshold Dec 21 15:12:57.461120069 (383h43m57.332203124s) [seg_pos={0 0}, upload=336789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:12:57.461220771 (383h43m57.332303816s) [upload=336789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:12:57.461227725 (383h43m57.33231077s) [upload=336789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:12:57.759106067 (383h43m57.630189122s) [upload=336789]: Done uploading segments... Dec 21 15:12:57.759131094 (383h43m57.630214139s) [upload=336789]: Upload finished. err: <nil> Duration: 30.003450675s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.X.............................................] duration: 29.769391049s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:13:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 29.769407882s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.187280431s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:13:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.187304477s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.940461603s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:12:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.940488736s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.X............................................] duration: 37.341579406s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:12:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 37.341608822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X.........X....X.X......XX............X..X] duration: 30.003037418s trace: https://jaeger.dev.storj.io/trace/5cbeb9b9c14e8fe4 FAIL (2025-12-21 15:12:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: G2JTQ3YVLZUGKNOBQ6B4WIYHYMNRUGOUTJDVWFYU435ETPEDZTTA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: BRQQYS45G5OQU4YBVSP5EUOVKNKGD74FDOZFIQBMHJ3AKZWYM6IQ) context deadline exceeded ======LOGS====== Dec 21 15:12:04.312612644 (383h43m4.183695679s) [upload=336778]: Starting upload Dec 21 15:12:04.312673389 (383h43m4.183756424s) [upload=336778]: Done starting upload Dec 21 15:12:04.312927423 (383h43m4.184010468s) [upload=336778]: Uploading segments... Dec 21 15:12:04.313091655 (383h43m4.18417471s) [upload=336778]: Got next segment. Inline: false Dec 21 15:12:04.313148504 (383h43m4.184231549s) [upload=336778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:12:04.313155737 (383h43m4.184238782s) [upload=336778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:12:04.560995340 (383h43m4.432078385s) [seg_pos={0 0}, upload=336778]: Begin upload segment... Dec 21 15:12:04.561063850 (383h43m4.432146906s) [seg_pos={0 0}, upload=336778]: Done begin upload segment. Dec 21 15:12:04.561099208 (383h43m4.432182253s) [noise=true , address=145.79.227.33:4870 , piece_id=C4UCC6LXMOQ2IQPTW3OYMFWX4RTOTO67L3B7SXRGIJM3BMP2MTZQ, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.561935941 (383h43m4.433018986s) [noise=true , address=77.90.31.10:4519 , piece_id=BRQQYS45G5OQU4YBVSP5EUOVKNKGD74FDOZFIQBMHJ3AKZWYM6IQ, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.562328318 (383h43m4.433411363s) [noise=true , address=145.79.227.10:4402 , piece_id=JCA2Q5ZR3E7JJWXLODYY6WVAKLLQJOHAXIYETOBFWJCC5EYXGS2Q, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.563322951 (383h43m4.434405986s) [noise=true , address=145.79.227.37:4565 , piece_id=YT6QPCV53QBQE4AJLAX6CV2JACFQOFNK2KTNFB2IQTUQDLDPHJDQ, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.563664041 (383h43m4.434747076s) [noise=true , address=145.79.227.52:4446 , piece_id=O3YOHSI76KS5E2KDNPOB75SWSX62KXSGC3JQVEQD6SUYP6X3XGTQ, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.564120490 (383h43m4.435203535s) [noise=true , address=145.79.227.51:4409 , piece_id=MGPU7M44JDXD332AC3SHXHDPFM6MGEMATIBCLPQZOUHFNC3LHYZQ, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.564542003 (383h43m4.435625038s) [noise=true , address=145.79.227.45:4230 , piece_id=T2DG7LDYLSDVAS5U2ZGPODA2RY5JQI24R2R7JLFLYO5GBFBLJ6OA, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.564925212 (383h43m4.436008257s) [noise=true , address=77.90.31.14:4664 , piece_id=R73MQRJBCQMHGRZJI7Z5XVWVOR3CLUL3ZQQNMIEP37WBOW3RFFWQ, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.565338499 (383h43m4.436421544s) [noise=true , address=145.79.227.49:4338 , piece_id=GCZHC64W5PZXEU77SU7HRYVMJ5LD7CKDCCKUMF5WR27UD7DLCA2Q, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.565591080 (383h43m4.436674125s) [noise=true , address=145.79.227.3:4151 , piece_id=QZBDJBHMTQZ3GQY7JWZD7B7EWDTAXV3EB2F4BBFVQV4TU36BTEJQ, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.565741768 (383h43m4.436824813s) [noise=true , address=145.79.227.11:4731 , piece_id=DGLRTXBYK6NCFI5XH5XAH4BXQUVVD7M4LRZO6HR6POPEBTEOKOVQ, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.566046297 (383h43m4.437129342s) [noise=true , address=77.90.31.17:4773 , piece_id=Y2Z2SH4EORQQYNIVTT3PBVJFMUG7KFT2ZYOUUR3XEKSVL24ZUHLA, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.566104808 (383h43m4.437187884s) [noise=true , address=77.90.31.18:4808 , piece_id=FAWK6MK3IOERUGRWJ2NC3ECFRLCCDNM4CWZW4M6Q2LFAIDYJYT2Q, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.566587067 (383h43m4.437670112s) [noise=true , address=45.91.116.12:4228 , piece_id=G2JTQ3YVLZUGKNOBQ6B4WIYHYMNRUGOUTJDVWFYU435ETPEDZTTA, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.566951410 (383h43m4.438034455s) [noise=true , address=145.79.227.43:4159 , piece_id=7NYLAIVAPXAY2HH4TIA7BAVZ7PKEKBWHQXV75IQ462KVPH2LNR6Q, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:04.567653948 (383h43m4.438736993s) [noise=true , address=45.91.116.6:4974 , piece_id=NYSUZNAIC6R5I3YPSV354IC6XTJ6OVCUY65KCW3FZT3DIVZSCQXA, seg_pos={0 0}, upload=336778]: Uploading piece... Dec 21 15:12:06.262833501 (383h43m6.133916546s) [upload=336778]: Next returned nil segment Dec 21 15:12:06.262853861 (383h43m6.133936906s) [upload=336778]: Waiting for error group managing segments... Dec 21 15:12:06.551438116 (383h43m6.422521161s) [noise=true , address=145.79.227.45:4230 , piece_id=T2DG7LDYLSDVAS5U2ZGPODA2RY5JQI24R2R7JLFLYO5GBFBLJ6OA, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.554114652 (383h43m6.425197698s) [noise=true , address=145.79.227.49:4338 , piece_id=GCZHC64W5PZXEU77SU7HRYVMJ5LD7CKDCCKUMF5WR27UD7DLCA2Q, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.558969346 (383h43m6.430052391s) [noise=true , address=145.79.227.33:4870 , piece_id=C4UCC6LXMOQ2IQPTW3OYMFWX4RTOTO67L3B7SXRGIJM3BMP2MTZQ, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.559792955 (383h43m6.430876s) [noise=true , address=145.79.227.51:4409 , piece_id=MGPU7M44JDXD332AC3SHXHDPFM6MGEMATIBCLPQZOUHFNC3LHYZQ, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.571813543 (383h43m6.442896588s) [noise=true , address=145.79.227.11:4731 , piece_id=DGLRTXBYK6NCFI5XH5XAH4BXQUVVD7M4LRZO6HR6POPEBTEOKOVQ, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.574600880 (383h43m6.445683925s) [noise=true , address=145.79.227.3:4151 , piece_id=QZBDJBHMTQZ3GQY7JWZD7B7EWDTAXV3EB2F4BBFVQV4TU36BTEJQ, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.577599801 (383h43m6.448682846s) [noise=true , address=145.79.227.52:4446 , piece_id=O3YOHSI76KS5E2KDNPOB75SWSX62KXSGC3JQVEQD6SUYP6X3XGTQ, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.582086174 (383h43m6.453169219s) [noise=true , address=145.79.227.10:4402 , piece_id=JCA2Q5ZR3E7JJWXLODYY6WVAKLLQJOHAXIYETOBFWJCC5EYXGS2Q, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.639140323 (383h43m6.510223369s) [noise=true , address=145.79.227.43:4159 , piece_id=7NYLAIVAPXAY2HH4TIA7BAVZ7PKEKBWHQXV75IQ462KVPH2LNR6Q, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.641202921 (383h43m6.512285966s) [noise=true , address=145.79.227.37:4565 , piece_id=YT6QPCV53QBQE4AJLAX6CV2JACFQOFNK2KTNFB2IQTUQDLDPHJDQ, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.733883547 (383h43m6.604966582s) [noise=true , address=77.90.31.14:4664 , piece_id=R73MQRJBCQMHGRZJI7Z5XVWVOR3CLUL3ZQQNMIEP37WBOW3RFFWQ, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.754686659 (383h43m6.625769704s) [noise=true , address=77.90.31.10:4519 , piece_id=BRQQYS45G5OQU4YBVSP5EUOVKNKGD74FDOZFIQBMHJ3AKZWYM6IQ, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.816800013 (383h43m6.687883058s) [noise=true , address=45.91.116.12:4228 , piece_id=G2JTQ3YVLZUGKNOBQ6B4WIYHYMNRUGOUTJDVWFYU435ETPEDZTTA, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.816818668 (383h43m6.687901713s) [seg_pos={0 0}, upload=336778]: Segment reached optimal threshold of 13 pieces. Dec 21 15:12:06.816997919 (383h43m6.688080954s) [noise=true , address=77.90.31.18:4808 , piece_id=FAWK6MK3IOERUGRWJ2NC3ECFRLCCDNM4CWZW4M6Q2LFAIDYJYT2Q, seg_pos={0 0}, upload=336778]: Done uploading piece. err: <nil> Dec 21 15:12:06.817065918 (383h43m6.688148953s) [noise=true , address=45.91.116.6:4974 , piece_id=NYSUZNAIC6R5I3YPSV354IC6XTJ6OVCUY65KCW3FZT3DIVZSCQXA, seg_pos={0 0}, upload=336778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:59880->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:59880->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59880->45.91.116.6:4974: use of closed network connection Dec 21 15:12:06.817075177 (383h43m6.688158222s) [noise=true , address=45.91.116.6:4974 , piece_id=NYSUZNAIC6R5I3YPSV354IC6XTJ6OVCUY65KCW3FZT3DIVZSCQXA, seg_pos={0 0}, upload=336778]: Long tail cancellation due to optimal threshold Dec 21 15:12:06.817259057 (383h43m6.688342102s) [noise=true , address=77.90.31.17:4773 , piece_id=Y2Z2SH4EORQQYNIVTT3PBVJFMUG7KFT2ZYOUUR3XEKSVL24ZUHLA, seg_pos={0 0}, upload=336778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 21 15:12:06.817272582 (383h43m6.688355627s) [noise=true , address=77.90.31.17:4773 , piece_id=Y2Z2SH4EORQQYNIVTT3PBVJFMUG7KFT2ZYOUUR3XEKSVL24ZUHLA, seg_pos={0 0}, upload=336778]: Long tail cancellation due to optimal threshold Dec 21 15:12:06.817294173 (383h43m6.688377218s) [seg_pos={0 0}, upload=336778]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 21 15:12:06.817380858 (383h43m6.688463913s) [upload=336778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:12:06.817389515 (383h43m6.68847256s) [upload=336778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:12:07.108964004 (383h43m6.980047039s) [upload=336778]: Done uploading segments... Dec 21 15:12:07.108981207 (383h43m6.980064242s) [upload=336778]: Upload finished. err: <nil> Duration: 30.003063186s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.020383115s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:12:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.020408593s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.515858322s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:11:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.515885074s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...................X.........X....X.X......XX............X..] duration: 25.448233616s trace: https://jaeger.dev.storj.io/trace/5ff6b969c46efa0a FAIL (2025-12-21 15:11:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: NFJ6WLEE3KK2WLZPCRBKBEUFMACT6DKVLFXKKDI5HRGO6HHQGKSA) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: 3DJHMYGHSCR4CIUU3P75CDLXHQ3UOM66MGGWSREGIR2CAXK7RD3A) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: IFEM3HKAWPYH535F2MQYVACWN5B753BYZAEGJTSTCBBILLYCPYUQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: HOLIJUKUVCPUOEVFSTTDKNC2ESNBESNJVYSXV4XVC44CQPR54MMA) context deadline exceeded ======LOGS====== Dec 21 15:10:07.063006666 (383h41m6.934089701s) [upload=336750]: Starting upload Dec 21 15:10:07.063064226 (383h41m6.934147261s) [upload=336750]: Done starting upload Dec 21 15:10:07.063168164 (383h41m6.934251209s) [upload=336750]: Uploading segments... Dec 21 15:10:07.063270208 (383h41m6.934353243s) [upload=336750]: Got next segment. Inline: false Dec 21 15:10:07.063320513 (383h41m6.934403558s) [upload=336750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:10:07.063325594 (383h41m6.934408639s) [upload=336750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:10:07.313015089 (383h41m7.184098134s) [seg_pos={0 0}, upload=336750]: Begin upload segment... Dec 21 15:10:07.313057790 (383h41m7.184140835s) [seg_pos={0 0}, upload=336750]: Done begin upload segment. Dec 21 15:10:07.313091474 (383h41m7.184174529s) [noise=true , address=45.91.116.7:4011 , piece_id=FDNFWOCCO4T45E4Z4TS7GPNATFXZBILQCVX25VAA2YONHYSRSXJA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.313135498 (383h41m7.184218543s) [noise=true , address=145.79.227.27:4691 , piece_id=PGN3Z52IQQEBRIIEOVDQOT6W5JBSPI4UA3V4FSDPRVBDCTQYORJA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.313743526 (383h41m7.184826561s) [noise=true , address=145.79.227.45:4229 , piece_id=WAPEGOTLYIJT7IS7I66ZGLCXJ3K7CBNP3ADPZ2IWCDVDKVWXMTXA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.314248036 (383h41m7.185331081s) [noise=true , address=145.79.227.38:4977 , piece_id=IFEM3HKAWPYH535F2MQYVACWN5B753BYZAEGJTSTCBBILLYCPYUQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.314653659 (383h41m7.185736704s) [noise=true , address=145.79.227.31:4797 , piece_id=CJVV7MDCJ2IMILVPGRSXJGXOZL3WFE3AXAYXEGYWXW7XS3AY4MVQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.315004497 (383h41m7.186087532s) [noise=true , address=77.90.31.19:4841 , piece_id=NBIOEZZ7XDDCTEAPVJBM5MBTGAS4EF7V5PWSIFRWP7R6GAMMYCFA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.315440507 (383h41m7.186523552s) [noise=true , address=145.79.227.46:4603 , piece_id=HOLIJUKUVCPUOEVFSTTDKNC2ESNBESNJVYSXV4XVC44CQPR54MMA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.315858563 (383h41m7.186941608s) [noise=true , address=77.90.31.17:4770 , piece_id=H5GPHHBI6B572QRW7RHHJDUJFQAOIQZWCQASIMPOCOPXAXBXHKZQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.316298862 (383h41m7.187381907s) [noise=true , address=145.79.227.49:4339 , piece_id=NFJ6WLEE3KK2WLZPCRBKBEUFMACT6DKVLFXKKDI5HRGO6HHQGKSA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.316807210 (383h41m7.187890245s) [noise=true , address=77.90.31.18:4805 , piece_id=6RJ7TDGCGSK3K6X2HENELN5CPOK5AOUF32UVZAVIWXWZUXSQXYJQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.318052571 (383h41m7.189135616s) [noise=true , address=45.91.116.5:4297 , piece_id=PPKCKJYHEUQS6TG6UDQVLCAILK44DBOKG43NTBU4CQ4XV3ZLXYVA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.318765368 (383h41m7.189848413s) [noise=true , address=45.91.116.11:4120 , piece_id=62T7CCSOTYSMG6KYGX2QTZINQLMZ3V4OH6WDL2FMPCJKPC5TQLDA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.319262665 (383h41m7.1903457s) [noise=true , address=77.90.31.11:4553 , piece_id=7UWY73YHVAE5OCWGJSJE73H52RTQQELCSHDFD7VRQYCI4EF43U3Q, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.319691161 (383h41m7.190774206s) [noise=true , address=145.79.227.51:4410 , piece_id=3DJHMYGHSCR4CIUU3P75CDLXHQ3UOM66MGGWSREGIR2CAXK7RD3A, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.320055305 (383h41m7.19113834s) [noise=true , address=145.79.227.28:4725 , piece_id=MHNKUGIRQVT34WVCULICXLD3P5U2I4YNCAO5DTDSV3J4AMX3BVCA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.320407876 (383h41m7.191490921s) [noise=true , address=145.79.227.48:4302 , piece_id=XYSRESLVHX4VYLD3KPCKLE3TAFFKGDDL2PBGE4JJ6S6ALVTIDOEA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:08.621250345 (383h41m8.49233341s) [noise=true , address=77.90.31.19:4841 , piece_id=NBIOEZZ7XDDCTEAPVJBM5MBTGAS4EF7V5PWSIFRWP7R6GAMMYCFA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 21 15:10:09.155381306 (383h41m9.026464341s) [upload=336750]: Next returned nil segment Dec 21 15:10:09.155401554 (383h41m9.026484629s) [upload=336750]: Waiting for error group managing segments... Dec 21 15:10:09.477513281 (383h41m9.348596316s) [noise=true , address=145.79.227.46:4603 , piece_id=HOLIJUKUVCPUOEVFSTTDKNC2ESNBESNJVYSXV4XVC44CQPR54MMA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.505621503 (383h41m9.376704548s) [noise=true , address=145.79.227.28:4725 , piece_id=MHNKUGIRQVT34WVCULICXLD3P5U2I4YNCAO5DTDSV3J4AMX3BVCA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.514327911 (383h41m9.385410956s) [noise=true , address=77.90.31.11:4553 , piece_id=7UWY73YHVAE5OCWGJSJE73H52RTQQELCSHDFD7VRQYCI4EF43U3Q, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.545121196 (383h41m9.416204241s) [noise=true , address=145.79.227.49:4339 , piece_id=NFJ6WLEE3KK2WLZPCRBKBEUFMACT6DKVLFXKKDI5HRGO6HHQGKSA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.555052657 (383h41m9.426135702s) [noise=true , address=145.79.227.38:4977 , piece_id=IFEM3HKAWPYH535F2MQYVACWN5B753BYZAEGJTSTCBBILLYCPYUQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.586521849 (383h41m9.457604894s) [noise=true , address=45.91.116.7:4011 , piece_id=FDNFWOCCO4T45E4Z4TS7GPNATFXZBILQCVX25VAA2YONHYSRSXJA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.587100510 (383h41m9.458183545s) [noise=true , address=45.91.116.11:4120 , piece_id=62T7CCSOTYSMG6KYGX2QTZINQLMZ3V4OH6WDL2FMPCJKPC5TQLDA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.596759724 (383h41m9.467842769s) [noise=true , address=77.90.31.17:4770 , piece_id=H5GPHHBI6B572QRW7RHHJDUJFQAOIQZWCQASIMPOCOPXAXBXHKZQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.597398670 (383h41m9.468481705s) [noise=true , address=45.91.116.5:4297 , piece_id=PPKCKJYHEUQS6TG6UDQVLCAILK44DBOKG43NTBU4CQ4XV3ZLXYVA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:10.259347544 (383h41m10.130430579s) [noise=true , address=145.79.227.45:4229 , piece_id=WAPEGOTLYIJT7IS7I66ZGLCXJ3K7CBNP3ADPZ2IWCDVDKVWXMTXA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:10.535825698 (383h41m10.406908743s) [noise=true , address=145.79.227.27:4691 , piece_id=PGN3Z52IQQEBRIIEOVDQOT6W5JBSPI4UA3V4FSDPRVBDCTQYORJA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:10.549036012 (383h41m10.420119047s) [noise=true , address=145.79.227.48:4302 , piece_id=XYSRESLVHX4VYLD3KPCKLE3TAFFKGDDL2PBGE4JJ6S6ALVTIDOEA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:11.022467444 (383h41m10.893550479s) [noise=true , address=145.79.227.51:4410 , piece_id=3DJHMYGHSCR4CIUU3P75CDLXHQ3UOM66MGGWSREGIR2CAXK7RD3A, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:11.022495107 (383h41m10.893578192s) [seg_pos={0 0}, upload=336750]: Segment reached optimal threshold of 13 pieces. Dec 21 15:10:11.022669830 (383h41m10.893752875s) [noise=true , address=77.90.31.18:4805 , piece_id=6RJ7TDGCGSK3K6X2HENELN5CPOK5AOUF32UVZAVIWXWZUXSQXYJQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:56654->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 21 15:10:11.022677404 (383h41m10.893760449s) [noise=true , address=77.90.31.18:4805 , piece_id=6RJ7TDGCGSK3K6X2HENELN5CPOK5AOUF32UVZAVIWXWZUXSQXYJQ, seg_pos={0 0}, upload=336750]: Long tail cancellation due to optimal threshold Dec 21 15:10:11.022879589 (383h41m10.893962634s) [noise=true , address=145.79.227.31:4797 , piece_id=CJVV7MDCJ2IMILVPGRSXJGXOZL3WFE3AXAYXEGYWXW7XS3AY4MVQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: read tcp 188.166.225.170:40520->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 15:10:11.022886182 (383h41m10.893969217s) [noise=true , address=145.79.227.31:4797 , piece_id=CJVV7MDCJ2IMILVPGRSXJGXOZL3WFE3AXAYXEGYWXW7XS3AY4MVQ, seg_pos={0 0}, upload=336750]: Long tail cancellation due to optimal threshold Dec 21 15:10:11.022906511 (383h41m10.893989556s) [seg_pos={0 0}, upload=336750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:10:11.022978848 (383h41m10.894061894s) [upload=336750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:10:11.022984008 (383h41m10.894067043s) [upload=336750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:10:11.322225335 (383h41m11.19330838s) [upload=336750]: Done uploading segments... Dec 21 15:10:11.322250403 (383h41m11.193333448s) [upload=336750]: Upload finished. err: <nil> Duration: 25.448259695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration: 32.413671244s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:11:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 32.413699358s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.86187079s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:11:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.861899987s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X...................X.........X....X.X......XX............X.] duration: 10.786832641s trace: https://jaeger.dev.storj.io/trace/5ff6b969c46efa0a FAIL (2025-12-21 15:10:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: NFJ6WLEE3KK2WLZPCRBKBEUFMACT6DKVLFXKKDI5HRGO6HHQGKSA) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: 3DJHMYGHSCR4CIUU3P75CDLXHQ3UOM66MGGWSREGIR2CAXK7RD3A) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: IFEM3HKAWPYH535F2MQYVACWN5B753BYZAEGJTSTCBBILLYCPYUQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: HOLIJUKUVCPUOEVFSTTDKNC2ESNBESNJVYSXV4XVC44CQPR54MMA) context deadline exceeded ======LOGS====== Dec 21 15:10:07.063006666 (383h41m6.934089701s) [upload=336750]: Starting upload Dec 21 15:10:07.063064226 (383h41m6.934147261s) [upload=336750]: Done starting upload Dec 21 15:10:07.063168164 (383h41m6.934251209s) [upload=336750]: Uploading segments... Dec 21 15:10:07.063270208 (383h41m6.934353243s) [upload=336750]: Got next segment. Inline: false Dec 21 15:10:07.063320513 (383h41m6.934403558s) [upload=336750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:10:07.063325594 (383h41m6.934408639s) [upload=336750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:10:07.313015089 (383h41m7.184098134s) [seg_pos={0 0}, upload=336750]: Begin upload segment... Dec 21 15:10:07.313057790 (383h41m7.184140835s) [seg_pos={0 0}, upload=336750]: Done begin upload segment. Dec 21 15:10:07.313091474 (383h41m7.184174529s) [noise=true , address=45.91.116.7:4011 , piece_id=FDNFWOCCO4T45E4Z4TS7GPNATFXZBILQCVX25VAA2YONHYSRSXJA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.313135498 (383h41m7.184218543s) [noise=true , address=145.79.227.27:4691 , piece_id=PGN3Z52IQQEBRIIEOVDQOT6W5JBSPI4UA3V4FSDPRVBDCTQYORJA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.313743526 (383h41m7.184826561s) [noise=true , address=145.79.227.45:4229 , piece_id=WAPEGOTLYIJT7IS7I66ZGLCXJ3K7CBNP3ADPZ2IWCDVDKVWXMTXA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.314248036 (383h41m7.185331081s) [noise=true , address=145.79.227.38:4977 , piece_id=IFEM3HKAWPYH535F2MQYVACWN5B753BYZAEGJTSTCBBILLYCPYUQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.314653659 (383h41m7.185736704s) [noise=true , address=145.79.227.31:4797 , piece_id=CJVV7MDCJ2IMILVPGRSXJGXOZL3WFE3AXAYXEGYWXW7XS3AY4MVQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.315004497 (383h41m7.186087532s) [noise=true , address=77.90.31.19:4841 , piece_id=NBIOEZZ7XDDCTEAPVJBM5MBTGAS4EF7V5PWSIFRWP7R6GAMMYCFA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.315440507 (383h41m7.186523552s) [noise=true , address=145.79.227.46:4603 , piece_id=HOLIJUKUVCPUOEVFSTTDKNC2ESNBESNJVYSXV4XVC44CQPR54MMA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.315858563 (383h41m7.186941608s) [noise=true , address=77.90.31.17:4770 , piece_id=H5GPHHBI6B572QRW7RHHJDUJFQAOIQZWCQASIMPOCOPXAXBXHKZQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.316298862 (383h41m7.187381907s) [noise=true , address=145.79.227.49:4339 , piece_id=NFJ6WLEE3KK2WLZPCRBKBEUFMACT6DKVLFXKKDI5HRGO6HHQGKSA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.316807210 (383h41m7.187890245s) [noise=true , address=77.90.31.18:4805 , piece_id=6RJ7TDGCGSK3K6X2HENELN5CPOK5AOUF32UVZAVIWXWZUXSQXYJQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.318052571 (383h41m7.189135616s) [noise=true , address=45.91.116.5:4297 , piece_id=PPKCKJYHEUQS6TG6UDQVLCAILK44DBOKG43NTBU4CQ4XV3ZLXYVA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.318765368 (383h41m7.189848413s) [noise=true , address=45.91.116.11:4120 , piece_id=62T7CCSOTYSMG6KYGX2QTZINQLMZ3V4OH6WDL2FMPCJKPC5TQLDA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.319262665 (383h41m7.1903457s) [noise=true , address=77.90.31.11:4553 , piece_id=7UWY73YHVAE5OCWGJSJE73H52RTQQELCSHDFD7VRQYCI4EF43U3Q, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.319691161 (383h41m7.190774206s) [noise=true , address=145.79.227.51:4410 , piece_id=3DJHMYGHSCR4CIUU3P75CDLXHQ3UOM66MGGWSREGIR2CAXK7RD3A, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.320055305 (383h41m7.19113834s) [noise=true , address=145.79.227.28:4725 , piece_id=MHNKUGIRQVT34WVCULICXLD3P5U2I4YNCAO5DTDSV3J4AMX3BVCA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.320407876 (383h41m7.191490921s) [noise=true , address=145.79.227.48:4302 , piece_id=XYSRESLVHX4VYLD3KPCKLE3TAFFKGDDL2PBGE4JJ6S6ALVTIDOEA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:08.621250345 (383h41m8.49233341s) [noise=true , address=77.90.31.19:4841 , piece_id=NBIOEZZ7XDDCTEAPVJBM5MBTGAS4EF7V5PWSIFRWP7R6GAMMYCFA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 21 15:10:09.155381306 (383h41m9.026464341s) [upload=336750]: Next returned nil segment Dec 21 15:10:09.155401554 (383h41m9.026484629s) [upload=336750]: Waiting for error group managing segments... Dec 21 15:10:09.477513281 (383h41m9.348596316s) [noise=true , address=145.79.227.46:4603 , piece_id=HOLIJUKUVCPUOEVFSTTDKNC2ESNBESNJVYSXV4XVC44CQPR54MMA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.505621503 (383h41m9.376704548s) [noise=true , address=145.79.227.28:4725 , piece_id=MHNKUGIRQVT34WVCULICXLD3P5U2I4YNCAO5DTDSV3J4AMX3BVCA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.514327911 (383h41m9.385410956s) [noise=true , address=77.90.31.11:4553 , piece_id=7UWY73YHVAE5OCWGJSJE73H52RTQQELCSHDFD7VRQYCI4EF43U3Q, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.545121196 (383h41m9.416204241s) [noise=true , address=145.79.227.49:4339 , piece_id=NFJ6WLEE3KK2WLZPCRBKBEUFMACT6DKVLFXKKDI5HRGO6HHQGKSA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.555052657 (383h41m9.426135702s) [noise=true , address=145.79.227.38:4977 , piece_id=IFEM3HKAWPYH535F2MQYVACWN5B753BYZAEGJTSTCBBILLYCPYUQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.586521849 (383h41m9.457604894s) [noise=true , address=45.91.116.7:4011 , piece_id=FDNFWOCCO4T45E4Z4TS7GPNATFXZBILQCVX25VAA2YONHYSRSXJA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.587100510 (383h41m9.458183545s) [noise=true , address=45.91.116.11:4120 , piece_id=62T7CCSOTYSMG6KYGX2QTZINQLMZ3V4OH6WDL2FMPCJKPC5TQLDA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.596759724 (383h41m9.467842769s) [noise=true , address=77.90.31.17:4770 , piece_id=H5GPHHBI6B572QRW7RHHJDUJFQAOIQZWCQASIMPOCOPXAXBXHKZQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.597398670 (383h41m9.468481705s) [noise=true , address=45.91.116.5:4297 , piece_id=PPKCKJYHEUQS6TG6UDQVLCAILK44DBOKG43NTBU4CQ4XV3ZLXYVA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:10.259347544 (383h41m10.130430579s) [noise=true , address=145.79.227.45:4229 , piece_id=WAPEGOTLYIJT7IS7I66ZGLCXJ3K7CBNP3ADPZ2IWCDVDKVWXMTXA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:10.535825698 (383h41m10.406908743s) [noise=true , address=145.79.227.27:4691 , piece_id=PGN3Z52IQQEBRIIEOVDQOT6W5JBSPI4UA3V4FSDPRVBDCTQYORJA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:10.549036012 (383h41m10.420119047s) [noise=true , address=145.79.227.48:4302 , piece_id=XYSRESLVHX4VYLD3KPCKLE3TAFFKGDDL2PBGE4JJ6S6ALVTIDOEA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:11.022467444 (383h41m10.893550479s) [noise=true , address=145.79.227.51:4410 , piece_id=3DJHMYGHSCR4CIUU3P75CDLXHQ3UOM66MGGWSREGIR2CAXK7RD3A, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:11.022495107 (383h41m10.893578192s) [seg_pos={0 0}, upload=336750]: Segment reached optimal threshold of 13 pieces. Dec 21 15:10:11.022669830 (383h41m10.893752875s) [noise=true , address=77.90.31.18:4805 , piece_id=6RJ7TDGCGSK3K6X2HENELN5CPOK5AOUF32UVZAVIWXWZUXSQXYJQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:56654->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 21 15:10:11.022677404 (383h41m10.893760449s) [noise=true , address=77.90.31.18:4805 , piece_id=6RJ7TDGCGSK3K6X2HENELN5CPOK5AOUF32UVZAVIWXWZUXSQXYJQ, seg_pos={0 0}, upload=336750]: Long tail cancellation due to optimal threshold Dec 21 15:10:11.022879589 (383h41m10.893962634s) [noise=true , address=145.79.227.31:4797 , piece_id=CJVV7MDCJ2IMILVPGRSXJGXOZL3WFE3AXAYXEGYWXW7XS3AY4MVQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: read tcp 188.166.225.170:40520->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 15:10:11.022886182 (383h41m10.893969217s) [noise=true , address=145.79.227.31:4797 , piece_id=CJVV7MDCJ2IMILVPGRSXJGXOZL3WFE3AXAYXEGYWXW7XS3AY4MVQ, seg_pos={0 0}, upload=336750]: Long tail cancellation due to optimal threshold Dec 21 15:10:11.022906511 (383h41m10.893989556s) [seg_pos={0 0}, upload=336750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:10:11.022978848 (383h41m10.894061894s) [upload=336750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:10:11.022984008 (383h41m10.894067043s) [upload=336750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:10:11.322225335 (383h41m11.19330838s) [upload=336750]: Done uploading segments... Dec 21 15:10:11.322250403 (383h41m11.193333448s) [upload=336750]: Upload finished. err: <nil> Duration: 10.78684779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X...................X.........X....X.X......XX............X] duration: 30.003488539s trace: https://jaeger.dev.storj.io/trace/5ff6b969c46efa0a FAIL (2025-12-21 15:10:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, Piece ID: NFJ6WLEE3KK2WLZPCRBKBEUFMACT6DKVLFXKKDI5HRGO6HHQGKSA) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: 3DJHMYGHSCR4CIUU3P75CDLXHQ3UOM66MGGWSREGIR2CAXK7RD3A) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: IFEM3HKAWPYH535F2MQYVACWN5B753BYZAEGJTSTCBBILLYCPYUQ) context deadline exceeded; piecestore: (Node ID: 1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, Piece ID: HOLIJUKUVCPUOEVFSTTDKNC2ESNBESNJVYSXV4XVC44CQPR54MMA) context deadline exceeded ======LOGS====== Dec 21 15:10:07.063006666 (383h41m6.934089701s) [upload=336750]: Starting upload Dec 21 15:10:07.063064226 (383h41m6.934147261s) [upload=336750]: Done starting upload Dec 21 15:10:07.063168164 (383h41m6.934251209s) [upload=336750]: Uploading segments... Dec 21 15:10:07.063270208 (383h41m6.934353243s) [upload=336750]: Got next segment. Inline: false Dec 21 15:10:07.063320513 (383h41m6.934403558s) [upload=336750]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:10:07.063325594 (383h41m6.934408639s) [upload=336750]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:10:07.313015089 (383h41m7.184098134s) [seg_pos={0 0}, upload=336750]: Begin upload segment... Dec 21 15:10:07.313057790 (383h41m7.184140835s) [seg_pos={0 0}, upload=336750]: Done begin upload segment. Dec 21 15:10:07.313091474 (383h41m7.184174529s) [noise=true , address=45.91.116.7:4011 , piece_id=FDNFWOCCO4T45E4Z4TS7GPNATFXZBILQCVX25VAA2YONHYSRSXJA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.313135498 (383h41m7.184218543s) [noise=true , address=145.79.227.27:4691 , piece_id=PGN3Z52IQQEBRIIEOVDQOT6W5JBSPI4UA3V4FSDPRVBDCTQYORJA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.313743526 (383h41m7.184826561s) [noise=true , address=145.79.227.45:4229 , piece_id=WAPEGOTLYIJT7IS7I66ZGLCXJ3K7CBNP3ADPZ2IWCDVDKVWXMTXA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.314248036 (383h41m7.185331081s) [noise=true , address=145.79.227.38:4977 , piece_id=IFEM3HKAWPYH535F2MQYVACWN5B753BYZAEGJTSTCBBILLYCPYUQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.314653659 (383h41m7.185736704s) [noise=true , address=145.79.227.31:4797 , piece_id=CJVV7MDCJ2IMILVPGRSXJGXOZL3WFE3AXAYXEGYWXW7XS3AY4MVQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.315004497 (383h41m7.186087532s) [noise=true , address=77.90.31.19:4841 , piece_id=NBIOEZZ7XDDCTEAPVJBM5MBTGAS4EF7V5PWSIFRWP7R6GAMMYCFA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.315440507 (383h41m7.186523552s) [noise=true , address=145.79.227.46:4603 , piece_id=HOLIJUKUVCPUOEVFSTTDKNC2ESNBESNJVYSXV4XVC44CQPR54MMA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.315858563 (383h41m7.186941608s) [noise=true , address=77.90.31.17:4770 , piece_id=H5GPHHBI6B572QRW7RHHJDUJFQAOIQZWCQASIMPOCOPXAXBXHKZQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.316298862 (383h41m7.187381907s) [noise=true , address=145.79.227.49:4339 , piece_id=NFJ6WLEE3KK2WLZPCRBKBEUFMACT6DKVLFXKKDI5HRGO6HHQGKSA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.316807210 (383h41m7.187890245s) [noise=true , address=77.90.31.18:4805 , piece_id=6RJ7TDGCGSK3K6X2HENELN5CPOK5AOUF32UVZAVIWXWZUXSQXYJQ, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.318052571 (383h41m7.189135616s) [noise=true , address=45.91.116.5:4297 , piece_id=PPKCKJYHEUQS6TG6UDQVLCAILK44DBOKG43NTBU4CQ4XV3ZLXYVA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.318765368 (383h41m7.189848413s) [noise=true , address=45.91.116.11:4120 , piece_id=62T7CCSOTYSMG6KYGX2QTZINQLMZ3V4OH6WDL2FMPCJKPC5TQLDA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.319262665 (383h41m7.1903457s) [noise=true , address=77.90.31.11:4553 , piece_id=7UWY73YHVAE5OCWGJSJE73H52RTQQELCSHDFD7VRQYCI4EF43U3Q, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.319691161 (383h41m7.190774206s) [noise=true , address=145.79.227.51:4410 , piece_id=3DJHMYGHSCR4CIUU3P75CDLXHQ3UOM66MGGWSREGIR2CAXK7RD3A, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.320055305 (383h41m7.19113834s) [noise=true , address=145.79.227.28:4725 , piece_id=MHNKUGIRQVT34WVCULICXLD3P5U2I4YNCAO5DTDSV3J4AMX3BVCA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:07.320407876 (383h41m7.191490921s) [noise=true , address=145.79.227.48:4302 , piece_id=XYSRESLVHX4VYLD3KPCKLE3TAFFKGDDL2PBGE4JJ6S6ALVTIDOEA, seg_pos={0 0}, upload=336750]: Uploading piece... Dec 21 15:10:08.621250345 (383h41m8.49233341s) [noise=true , address=77.90.31.19:4841 , piece_id=NBIOEZZ7XDDCTEAPVJBM5MBTGAS4EF7V5PWSIFRWP7R6GAMMYCFA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 21 15:10:09.155381306 (383h41m9.026464341s) [upload=336750]: Next returned nil segment Dec 21 15:10:09.155401554 (383h41m9.026484629s) [upload=336750]: Waiting for error group managing segments... Dec 21 15:10:09.477513281 (383h41m9.348596316s) [noise=true , address=145.79.227.46:4603 , piece_id=HOLIJUKUVCPUOEVFSTTDKNC2ESNBESNJVYSXV4XVC44CQPR54MMA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.505621503 (383h41m9.376704548s) [noise=true , address=145.79.227.28:4725 , piece_id=MHNKUGIRQVT34WVCULICXLD3P5U2I4YNCAO5DTDSV3J4AMX3BVCA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.514327911 (383h41m9.385410956s) [noise=true , address=77.90.31.11:4553 , piece_id=7UWY73YHVAE5OCWGJSJE73H52RTQQELCSHDFD7VRQYCI4EF43U3Q, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.545121196 (383h41m9.416204241s) [noise=true , address=145.79.227.49:4339 , piece_id=NFJ6WLEE3KK2WLZPCRBKBEUFMACT6DKVLFXKKDI5HRGO6HHQGKSA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.555052657 (383h41m9.426135702s) [noise=true , address=145.79.227.38:4977 , piece_id=IFEM3HKAWPYH535F2MQYVACWN5B753BYZAEGJTSTCBBILLYCPYUQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.586521849 (383h41m9.457604894s) [noise=true , address=45.91.116.7:4011 , piece_id=FDNFWOCCO4T45E4Z4TS7GPNATFXZBILQCVX25VAA2YONHYSRSXJA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.587100510 (383h41m9.458183545s) [noise=true , address=45.91.116.11:4120 , piece_id=62T7CCSOTYSMG6KYGX2QTZINQLMZ3V4OH6WDL2FMPCJKPC5TQLDA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.596759724 (383h41m9.467842769s) [noise=true , address=77.90.31.17:4770 , piece_id=H5GPHHBI6B572QRW7RHHJDUJFQAOIQZWCQASIMPOCOPXAXBXHKZQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:09.597398670 (383h41m9.468481705s) [noise=true , address=45.91.116.5:4297 , piece_id=PPKCKJYHEUQS6TG6UDQVLCAILK44DBOKG43NTBU4CQ4XV3ZLXYVA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:10.259347544 (383h41m10.130430579s) [noise=true , address=145.79.227.45:4229 , piece_id=WAPEGOTLYIJT7IS7I66ZGLCXJ3K7CBNP3ADPZ2IWCDVDKVWXMTXA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:10.535825698 (383h41m10.406908743s) [noise=true , address=145.79.227.27:4691 , piece_id=PGN3Z52IQQEBRIIEOVDQOT6W5JBSPI4UA3V4FSDPRVBDCTQYORJA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:10.549036012 (383h41m10.420119047s) [noise=true , address=145.79.227.48:4302 , piece_id=XYSRESLVHX4VYLD3KPCKLE3TAFFKGDDL2PBGE4JJ6S6ALVTIDOEA, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:11.022467444 (383h41m10.893550479s) [noise=true , address=145.79.227.51:4410 , piece_id=3DJHMYGHSCR4CIUU3P75CDLXHQ3UOM66MGGWSREGIR2CAXK7RD3A, seg_pos={0 0}, upload=336750]: Done uploading piece. err: <nil> Dec 21 15:10:11.022495107 (383h41m10.893578192s) [seg_pos={0 0}, upload=336750]: Segment reached optimal threshold of 13 pieces. Dec 21 15:10:11.022669830 (383h41m10.893752875s) [noise=true , address=77.90.31.18:4805 , piece_id=6RJ7TDGCGSK3K6X2HENELN5CPOK5AOUF32UVZAVIWXWZUXSQXYJQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:56654->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 21 15:10:11.022677404 (383h41m10.893760449s) [noise=true , address=77.90.31.18:4805 , piece_id=6RJ7TDGCGSK3K6X2HENELN5CPOK5AOUF32UVZAVIWXWZUXSQXYJQ, seg_pos={0 0}, upload=336750]: Long tail cancellation due to optimal threshold Dec 21 15:10:11.022879589 (383h41m10.893962634s) [noise=true , address=145.79.227.31:4797 , piece_id=CJVV7MDCJ2IMILVPGRSXJGXOZL3WFE3AXAYXEGYWXW7XS3AY4MVQ, seg_pos={0 0}, upload=336750]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: read tcp 188.166.225.170:40520->145.79.227.31:4797: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 21 15:10:11.022886182 (383h41m10.893969217s) [noise=true , address=145.79.227.31:4797 , piece_id=CJVV7MDCJ2IMILVPGRSXJGXOZL3WFE3AXAYXEGYWXW7XS3AY4MVQ, seg_pos={0 0}, upload=336750]: Long tail cancellation due to optimal threshold Dec 21 15:10:11.022906511 (383h41m10.893989556s) [seg_pos={0 0}, upload=336750]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:10:11.022978848 (383h41m10.894061894s) [upload=336750]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:10:11.022984008 (383h41m10.894067043s) [upload=336750]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:10:11.322225335 (383h41m11.19330838s) [upload=336750]: Done uploading segments... Dec 21 15:10:11.322250403 (383h41m11.193333448s) [upload=336750]: Upload finished. err: <nil> Duration: 30.003514659s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 7.377142964s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:10:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.377169755s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration: 13.334868264s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:09:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 13.334898142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...................X.........X....X.X......XX............] duration: 5.813937698s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:09:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 5.813959279s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.059000426s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:09:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.059027828s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.X.........................................] duration: 26.327560321s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:09:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 26.327599145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.016992057s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:08:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.01701999s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X...................X.........X....X.X......XX...........] duration: 6.02860004s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:08:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 6.028626542s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.779411871s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:08:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.779490312s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.X........................................] duration: 23.648769269s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:07:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 23.6487912s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X...................X.........X....X.X......XX..........] duration: 6.026810412s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:07:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 6.026840109s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.578264181s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:07:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.578314155s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.X.......................................] duration: 14.225195152s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:06:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 14.225228266s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.250509627s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:06:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.250541057s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X...................X.........X....X.X......XX.........] duration: 11.165489498s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:06:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 11.165519013s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.440284195s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:05:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.440307809s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 10.9360633s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:05:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 10.936091833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X...................X.........X....X.X......XX........] duration: 5.757366174s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:05:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 5.757393446s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.322179085s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:05:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.322208761s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 15.486610432s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:05:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 15.486642151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X......X...................X.........X....X.X......XX.......] duration: 6.049996072s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:05:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 6.050023645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X......X...................X.........X....X.X......XX......] duration: 24.872139272s trace: https://jaeger.dev.storj.io/trace/262dee68480330ec FAIL (2025-12-21 15:04:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ) context deadline exceeded ======LOGS====== Dec 21 15:00:18.429413553 (383h31m18.300496598s) [upload=336605]: Starting upload Dec 21 15:00:18.429515016 (383h31m18.300598081s) [upload=336605]: Done starting upload Dec 21 15:00:18.429917874 (383h31m18.301000919s) [upload=336605]: Uploading segments... Dec 21 15:00:18.430008195 (383h31m18.30109127s) [upload=336605]: Got next segment. Inline: false Dec 21 15:00:18.430049924 (383h31m18.301132969s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 15:00:18.430059413 (383h31m18.301142458s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 15:00:18.945606666 (383h31m18.816689711s) [seg_pos={0 0}, upload=336605]: Begin upload segment... Dec 21 15:00:18.945662433 (383h31m18.816745468s) [seg_pos={0 0}, upload=336605]: Done begin upload segment. Dec 21 15:00:18.945685406 (383h31m18.816768451s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945716445 (383h31m18.81679949s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.945832596 (383h31m18.816915641s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946869971 (383h31m18.817953016s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.946978287 (383h31m18.818061332s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947270284 (383h31m18.818353329s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947636200 (383h31m18.818719245s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.947779814 (383h31m18.818862859s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948235693 (383h31m18.819318738s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948437246 (383h31m18.819520291s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.948839352 (383h31m18.819922387s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949312582 (383h31m18.820395627s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.949727212 (383h31m18.820810257s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950034778 (383h31m18.821117863s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.950115542 (383h31m18.821198587s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:18.951013672 (383h31m18.822096727s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Uploading piece... Dec 21 15:00:20.745213292 (383h31m20.616296337s) [upload=336605]: Next returned nil segment Dec 21 15:00:20.745231556 (383h31m20.616314601s) [upload=336605]: Waiting for error group managing segments... Dec 21 15:00:21.181038025 (383h31m21.05212107s) [noise=true , address=145.79.227.36:4518 , piece_id=GN2PH3NZL3YG4D62M4QLWSYUPUGCTSPSAZTAUE7432YW3Z46PLQA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.222546242 (383h31m21.093629277s) [noise=true , address=77.90.31.19:4844 , piece_id=C2W7AFXHGCNTZR44DN5QH33WHWPQKS2UFDFSYOHCQA5RBFR6VFEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.251190745 (383h31m21.12227379s) [noise=true , address=45.91.116.5:4299 , piece_id=OEZ7QRSYZS6KCZTMUSEMGBYCYKHFUO6YSP6BQQMUOENMHHMEMMYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.294368502 (383h31m21.165451537s) [noise=true , address=145.79.227.27:4690 , piece_id=533CZXCWFXM3LZKJYFWOM5ZHY5TX2Z37DHTBJFXFNWMPIPBYOMOA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.382585389 (383h31m21.253668434s) [noise=true , address=45.91.116.10:4083 , piece_id=KWYZRJLDEC4QFGAOU4MTC2MJJOH4NWE5WPDZZCZQB7K3YL3MXAYA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.445236846 (383h31m21.316319891s) [noise=true , address=145.79.227.52:4445 , piece_id=RO5EUMWIURC5P7ZYWZTFWITPCSUWHOIGV3LH7GKE7QV3UZQWD7EQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.495619973 (383h31m21.366703018s) [noise=true , address=145.79.227.47:4266 , piece_id=ACZVH2AYRAWQ6DOJZT5KNPASQLTYBGFJSUVHNDMY3VVLC4I6FKEQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.537959062 (383h31m21.409042107s) [noise=true , address=145.79.227.17:4972 , piece_id=E5TPVFI5LCLKHFNCZLNINUDAFLPL4Z66TRVX5YSBBCI6XXML5WJA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.594515644 (383h31m21.465598679s) [noise=true , address=145.79.227.50:4374 , piece_id=JX2HLFXEQ4TU5QZOORJLM4ODY5TJGYBKOLHIT5EOL4O2VFHNI6PA, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.804032023 (383h31m21.675115068s) [noise=true , address=145.79.227.32:4834 , piece_id=N3BYEM7IF4K4LVCHC7VWRM6XPZJ6XAUQPE7YK2G7Z2GT6K7XIYGQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.823062991 (383h31m21.694146036s) [noise=true , address=145.79.227.48:4303 , piece_id=JISFULFPWUASUC2VYDJRZAPVGGN2VM2NNBPJG53W64STY4T2KQ7A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.908261942 (383h31m21.779344977s) [noise=true , address=145.79.227.46:4603 , piece_id=GGF2TPHSB33RBABBP5RTWUPMRD6H5LD4WCHJXUIZSB747HWPOF6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918341344 (383h31m21.789424389s) [noise=true , address=145.79.227.45:4229 , piece_id=TIOB373L7DPA3RQ7LCGKJ6RVYNPUORJWOGEI75X5SVAIBX7K2XNQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: <nil> Dec 21 15:00:21.918384055 (383h31m21.7894671s) [seg_pos={0 0}, upload=336605]: Segment reached optimal threshold of 13 pieces. Dec 21 15:00:21.918679979 (383h31m21.789763024s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918688696 (383h31m21.789771731s) [noise=true , address=145.79.227.15:4874 , piece_id=5LG2NPEWSZCDEK634LKTJ6JJJSWJI3SAGWDVDPUK2NGTFJRM5Q5Q, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.918976373 (383h31m21.790059418s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.918984398 (383h31m21.790067443s) [noise=true , address=145.79.227.41:4085 , piece_id=GR3OSDL7FSJ22Y7NRAGQ3PBIW7YWMHBOAWBKM45Q7A3W47IL6WVQ, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919124996 (383h31m21.790208041s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1): protocol: expected piece hash; context canceled; EOF Dec 21 15:00:21.919131048 (383h31m21.790214093s) [noise=true , address=145.79.227.10:4401 , piece_id=TKR2Y53JJ6JFU7M5L5FYX4JMXCUJR22MW6EEODI3FCYHZ4NMJM6A, seg_pos={0 0}, upload=336605]: Long tail cancellation due to optimal threshold Dec 21 15:00:21.919155254 (383h31m21.790238299s) [seg_pos={0 0}, upload=336605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 15:00:21.919347320 (383h31m21.790430365s) [upload=336605]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 15:00:21.919369682 (383h31m21.790452727s) [upload=336605]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 15:00:22.244137197 (383h31m22.115220242s) [upload=336605]: Done uploading segments... Dec 21 15:00:22.244157465 (383h31m22.11524051s) [upload=336605]: Upload finished. err: <nil> Duration: 24.872165463s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.722505463s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:04:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.722534747s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 17.932735876s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 17.932762266s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.300212179s trace: https://jaeger.dev.storj.io/trace/50da946ad1f41132 FAIL (2025-12-21 15:03:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.300238799s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 21.556690509s trace: https://jaeger.dev.storj.io/trace/3b34a8e18a35b4ce FAIL (2025-12-21 15:03:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu, Piece ID: PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ) context deadline exceeded; piecestore: (Node ID: 12HCCmBhKtqUembEH7kRqTjViQ8Kj6WLGtmcwrtcWh1HUVrXB4D, Piece ID: VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A) context deadline exceeded; piecestore: (Node ID: 12wWhCQfZ1Eg64FvirY1QLikYy8X31Vkk87NvZToVn4utw3Kxai, Piece ID: PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ) context deadline exceeded; piecestore: (Node ID: 12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, Piece ID: RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q) context deadline exceeded; piecestore: (Node ID: 1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, Piece ID: XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA) context deadline exceeded; piecestore: (Node ID: 1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, Piece ID: BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA) context deadline exceeded ======LOGS====== RJ4LA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.746971901 (383h8m35.618054936s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747057333 (383h8m35.618140368s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747504104 (383h8m35.618587139s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747873047 (383h8m35.618956122s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.747885811 (383h8m35.618968846s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748231148 (383h8m35.619314183s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748630238 (383h8m35.619713273s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.748642502 (383h8m35.619725537s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749088952 (383h8m35.620171987s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.749456061 (383h8m35.620539106s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.750100939 (383h8m35.621183974s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751117965 (383h8m35.62220101s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.751669325 (383h8m35.62275237s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.755386584 (383h8m35.626469629s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:35.757366543 (383h8m35.628449588s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Uploading piece... Dec 21 14:37:37.292702900 (383h8m37.163785946s) [upload=336266]: Next returned nil segment Dec 21 14:37:37.292716456 (383h8m37.163799501s) [upload=336266]: Waiting for error group managing segments... Dec 21 14:37:37.688209899 (383h8m37.559292945s) [noise=true , address=145.79.227.40:4068 , piece_id=RFMNLQ5H2IRQDLAFNTSV273K657ZEHRBK4ZN4JC46ZQLYGURJ4LA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.873040678 (383h8m37.744123713s) [noise=true , address=45.91.116.10:4110 , piece_id=2W3NUKUFS6WIL576IFI7OPAJSUVHOYMNW64BEIIOUQ4V43LGKL3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.908398705 (383h8m37.77948174s) [noise=true , address=45.91.116.7:4025 , piece_id=BFSOBVRBAJTRCQVX3F75VVRMNVJ4B36AVUN7XRQJABNNX7KN2UKA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.913105538 (383h8m37.784188573s) [noise=true , address=45.91.116.6:4004 , piece_id=KMRNXDYJJO5WOZCI2ODG5HSMK67IQNW5T5VP53B6OMC56DVNCD4Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:37.923940999 (383h8m37.795024034s) [noise=true , address=45.91.116.11:4127 , piece_id=ZKBGFHML7WBNQETKEB27SDLXVH3MFZPUGBIPIT2Z2U4CC3K4NOJA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.194411519 (383h8m38.065494554s) [noise=true , address=45.91.116.9:4286 , piece_id=YYCOEK3PYBTYB754JHISVKAVDUB7F3EW2HRBNFTCE6EUZMPGE2QA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.527806087 (383h8m38.398889142s) [noise=true , address=45.91.116.13:4160 , piece_id=RSRZELJVA5IBKTYMA6O5PEDUB3VJHJRYTIG6GG6CRWIFZLTUR37Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:38.646580882 (383h8m38.517663917s) [noise=true , address=45.91.116.5:4307 , piece_id=BGGSIKAOWTWX4SD4UEOQQGGQAKC6HP463NODIDLQYB73INY63EWA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:39.323990179 (383h8m39.195073224s) [noise=true , address=45.91.116.14:4217 , piece_id=Q2WKG2SE7R532ESYUUNSNCWZYDDYNC3K4UHGMQSFAD3AGBHSTMFQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:42.148190200 (383h8m42.019273245s) [noise=true , address=77.90.31.12:4608 , piece_id=PBXQUO7XGSWNCTGS5L4TMTCCAJ2GI55CJEAIHYHJOV4UJIONLZKQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:43.527833323 (383h8m43.398916368s) [noise=true , address=77.90.31.18:4833 , piece_id=XF5CIVFS4IOELKZHA45LMNF7ULZIBMPE2ER4T5GQB2NRDMATSIHA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:44.592615599 (383h8m44.463698644s) [noise=true , address=77.90.31.11:4580 , piece_id=VLZVW5UJI2R77LDML23IXXD5JBJSQKPJDQUDGLSG5PS57VKIZA3A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:47.562760933 (383h8m47.433843978s) [noise=true , address=77.90.31.14:4674 , piece_id=PYZRFBVJIWAZ7UDVWRZFHQAJ7JKLE3HMKKSKWNGGVH2S5YC3E6ZQ, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095443016 (383h8m48.966526051s) [noise=true , address=77.90.31.16:4738 , piece_id=XBRWTCTBKNKTYD22WRE65CGJ35YK3KZ5IM27EPAEKEI3FFMKQXYA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: <nil> Dec 21 14:37:49.095467964 (383h8m48.966550999s) [seg_pos={0 0}, upload=336266]: Segment reached optimal threshold of 14 pieces. Dec 21 14:37:49.095669577 (383h8m48.966752622s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZPTKQVuPNRTddMCCVpEgu9z5aC3ooJC854HpuA9223XEQA1o): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095677102 (383h8m48.966760137s) [noise=true , address=77.90.31.13:4651 , piece_id=5PO5W3HTS7CYRTDHCQPCMAACK7KGQQMAE7QKO2SSN2PYINRHJF4A, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095884917 (383h8m48.966967962s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x): protocol: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60962->77.90.31.21:4897: use of closed network connection Dec 21 14:37:49.095894215 (383h8m48.96697726s) [noise=true , address=77.90.31.21:4897 , piece_id=65YSSO4VG3B4BIT7EOGI3YLP63TIAUC3PGVFFLXPHQJNXSCPYSTA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.095908392 (383h8m48.966991467s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv): protocol: expected piece hash; context canceled; EOF Dec 21 14:37:49.095932698 (383h8m48.967015753s) [noise=true , address=77.90.31.19:4847 , piece_id=EMEN4SDKG6YZ7QFT6W447EGUIS53QFJETKJDZRWDHHUL3JX6DK5Q, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096134232 (383h8m48.967217267s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: expected piece hash; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53326->64.49.13.169:4716: use of closed network connection; EOF Dec 21 14:37:49.096146565 (383h8m48.9672296s) [noise=true , address=64.49.13.169:4716 , piece_id=OBRBUHWSTDTMIOYPGBP5MENHEDJPWTYLQHBN3OJIEXXGPIX4IYFA, seg_pos={0 0}, upload=336266]: Long tail cancellation due to optimal threshold Dec 21 14:37:49.096187092 (383h8m48.967270127s) [seg_pos={0 0}, upload=336266]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 21 14:37:49.096271955 (383h8m48.967355s) [upload=336266]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 14:37:49.096278827 (383h8m48.967361873s) [upload=336266]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 14:37:49.399124265 (383h8m49.2702073s) [upload=336266]: Done uploading segments... Dec 21 14:37:49.399150976 (383h8m49.270234031s) [upload=336266]: Upload finished. err: <nil> Duration: 21.556730645s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)