Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.....X.....X...............................] duration: 25.708984496s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:17:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 25.709005906s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X.............X...........X............X.X....X........X..] duration: 3.485559918s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:17:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.48558747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.XXX...X.XX] duration: 30.002735294s trace: https://jaeger.dev.storj.io/trace/28cb2815058e93d FAIL (2025-12-30 05:16:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: NAOWHWNPRTTAWLKMDQGKMQ3OQJLL7MYV6P6Y26JKP3XIZCR42WDQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: A2MFUK4NWQL5AAW6H5ROOE7TBVMPAHNGVBACPD5BM2TBNG27D4TQ) context deadline exceeded ======LOGS====== egments... Dec 30 05:16:20.724594703 (589h47m20.595677748s) [upload=517022]: Got next segment. Inline: false Dec 30 05:16:20.724661430 (589h47m20.595744465s) [upload=517022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:16:20.724667842 (589h47m20.595750887s) [upload=517022]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:16:20.971084969 (589h47m20.842168024s) [seg_pos={0 0}, upload=517022]: Begin upload segment... Dec 30 05:16:20.971142690 (589h47m20.842225725s) [seg_pos={0 0}, upload=517022]: Done begin upload segment. Dec 30 05:16:20.971176614 (589h47m20.842259649s) [noise=true , address=145.79.227.4:4185 , piece_id=GDVNMN7AV3635LCYD7366GGDUJCT47EXUGLIJIWPJJLJV4GWKAJA, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.971300811 (589h47m20.842383856s) [noise=true , address=45.91.116.14:4190 , piece_id=KSMDOFLHHFMNWWUOY7GIJXNGQYDUUUMRIODAKTMUMKCRMMUPBAZA, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.971915611 (589h47m20.842998656s) [noise=true , address=145.79.227.20:4439 , piece_id=VTMQN4EQ7C4YBT2KZDZC737G5C3IFMC5NBH5CKNLL5BLBTTICSXQ, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.972333697 (589h47m20.843416732s) [noise=true , address=145.79.227.15:4874 , piece_id=HFRDJJLRMWIW6B2PINDK2PUPVUYN2A2HVQGM7OHBKGQTW5LGEWDA, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.972816236 (589h47m20.843899281s) [noise=true , address=145.79.227.16:4910 , piece_id=G5VDSM7PGPLIFG5M3L5K52YJIV3TH4KTQQCW3MWAPCR73C7TD34Q, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.972834752 (589h47m20.843917797s) [noise=true , address=45.91.116.12:4226 , piece_id=EMG3MXE6B4VE7OMN4RYWAXPID5FMTFSYZZJGLSUUOOFTL6TA54RQ, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.973278256 (589h47m20.844361301s) [noise=true , address=145.79.227.31:4799 , piece_id=L5FINSC2XH54KKNBOFM2VEPGY5CWUPTBGMLSRNULXFKVLRFSG6KQ, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.973305398 (589h47m20.844388443s) [noise=true , address=145.79.227.28:4726 , piece_id=4W7ETRY5L56O2JOXQU2L42CMVBPPWB5U4QOGGMSZ5KRHSWX7WV5A, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.973743312 (589h47m20.844826357s) [noise=true , address=77.90.31.13:4626 , piece_id=NAOWHWNPRTTAWLKMDQGKMQ3OQJLL7MYV6P6Y26JKP3XIZCR42WDQ, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.973773309 (589h47m20.844856364s) [noise=true , address=77.90.31.14:4663 , piece_id=QB6J2XSJIJKST3TMID4FUTAL4RRCTC3GNYAVLKVAUZ76ZJ627THQ, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.974223375 (589h47m20.84530642s) [noise=true , address=145.79.227.25:4619 , piece_id=ZKAMGSKNYKYIC7IE3574MIZ664YHIE2S44IMK6L4BPUVBQQSSBRQ, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.974356278 (589h47m20.845439313s) [noise=true , address=145.79.227.41:4085 , piece_id=6UE6ANLQTAG7523X36SGMXIQGA6IWXZZ4UIZD6WBKHBPQGSSYQEA, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.974838447 (589h47m20.845921492s) [noise=true , address=45.91.116.5:4298 , piece_id=A2MFUK4NWQL5AAW6H5ROOE7TBVMPAHNGVBACPD5BM2TBNG27D4TQ, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.975301198 (589h47m20.846384233s) [noise=true , address=145.79.227.19:4077 , piece_id=NYOPON6JAI22E47SYTB5SFK3ZCH25LME3YI3HJND2GPUEKBQKRYQ, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.975780861 (589h47m20.846863926s) [noise=true , address=145.79.227.23:4547 , piece_id=ME5HWG2UGMNCMCDU22UXCC23ARMO352VFCT3IXSKPR6HEXZQFN6Q, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:20.977763216 (589h47m20.848846261s) [noise=true , address=145.79.227.6:4258 , piece_id=DUZCGNI5D3ONOAKGLLCNYFF2IXEWAOWP4W2LEJZBO4PNNFJQ5DRQ, seg_pos={0 0}, upload=517022]: Uploading piece... Dec 30 05:16:22.643393575 (589h47m22.51447663s) [upload=517022]: Next returned nil segment Dec 30 05:16:22.643414264 (589h47m22.514497299s) [upload=517022]: Waiting for error group managing segments... Dec 30 05:16:22.950618049 (589h47m22.821701095s) [noise=true , address=145.79.227.15:4874 , piece_id=HFRDJJLRMWIW6B2PINDK2PUPVUYN2A2HVQGM7OHBKGQTW5LGEWDA, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:22.962853057 (589h47m22.833936102s) [noise=true , address=145.79.227.31:4799 , piece_id=L5FINSC2XH54KKNBOFM2VEPGY5CWUPTBGMLSRNULXFKVLRFSG6KQ, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:22.970552348 (589h47m22.841635393s) [noise=true , address=145.79.227.23:4547 , piece_id=ME5HWG2UGMNCMCDU22UXCC23ARMO352VFCT3IXSKPR6HEXZQFN6Q, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:22.971936844 (589h47m22.843019889s) [noise=true , address=145.79.227.28:4726 , piece_id=4W7ETRY5L56O2JOXQU2L42CMVBPPWB5U4QOGGMSZ5KRHSWX7WV5A, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:22.974434069 (589h47m22.845517104s) [noise=true , address=145.79.227.6:4258 , piece_id=DUZCGNI5D3ONOAKGLLCNYFF2IXEWAOWP4W2LEJZBO4PNNFJQ5DRQ, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:22.974615795 (589h47m22.84569884s) [noise=true , address=145.79.227.20:4439 , piece_id=VTMQN4EQ7C4YBT2KZDZC737G5C3IFMC5NBH5CKNLL5BLBTTICSXQ, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:22.979209662 (589h47m22.850292698s) [noise=true , address=145.79.227.4:4185 , piece_id=GDVNMN7AV3635LCYD7366GGDUJCT47EXUGLIJIWPJJLJV4GWKAJA, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:22.979228337 (589h47m22.850311383s) [noise=true , address=145.79.227.41:4085 , piece_id=6UE6ANLQTAG7523X36SGMXIQGA6IWXZZ4UIZD6WBKHBPQGSSYQEA, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:22.986895708 (589h47m22.857978753s) [noise=true , address=145.79.227.19:4077 , piece_id=NYOPON6JAI22E47SYTB5SFK3ZCH25LME3YI3HJND2GPUEKBQKRYQ, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:22.992896334 (589h47m22.863979369s) [noise=true , address=145.79.227.16:4910 , piece_id=G5VDSM7PGPLIFG5M3L5K52YJIV3TH4KTQQCW3MWAPCR73C7TD34Q, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:23.009218174 (589h47m22.880301219s) [noise=true , address=145.79.227.25:4619 , piece_id=ZKAMGSKNYKYIC7IE3574MIZ664YHIE2S44IMK6L4BPUVBQQSSBRQ, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:23.160496793 (589h47m23.031579838s) [noise=true , address=77.90.31.13:4626 , piece_id=NAOWHWNPRTTAWLKMDQGKMQ3OQJLL7MYV6P6Y26JKP3XIZCR42WDQ, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:23.313042825 (589h47m23.18412587s) [noise=true , address=45.91.116.5:4298 , piece_id=A2MFUK4NWQL5AAW6H5ROOE7TBVMPAHNGVBACPD5BM2TBNG27D4TQ, seg_pos={0 0}, upload=517022]: Done uploading piece. err: <nil> Dec 30 05:16:23.313063815 (589h47m23.18414685s) [seg_pos={0 0}, upload=517022]: Segment reached optimal threshold of 13 pieces. Dec 30 05:16:23.313246402 (589h47m23.184329437s) [noise=true , address=45.91.116.12:4226 , piece_id=EMG3MXE6B4VE7OMN4RYWAXPID5FMTFSYZZJGLSUUOOFTL6TA54RQ, seg_pos={0 0}, upload=517022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:52106->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:52106->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52106->45.91.116.12:4226: use of closed network connection Dec 30 05:16:23.313252955 (589h47m23.184336s) [noise=true , address=45.91.116.12:4226 , piece_id=EMG3MXE6B4VE7OMN4RYWAXPID5FMTFSYZZJGLSUUOOFTL6TA54RQ, seg_pos={0 0}, upload=517022]: Long tail cancellation due to optimal threshold Dec 30 05:16:23.313481931 (589h47m23.184564966s) [noise=true , address=45.91.116.14:4190 , piece_id=KSMDOFLHHFMNWWUOY7GIJXNGQYDUUUMRIODAKTMUMKCRMMUPBAZA, seg_pos={0 0}, upload=517022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:46526->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:46526->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46526->45.91.116.14:4190: use of closed network connection Dec 30 05:16:23.313493122 (589h47m23.184576157s) [noise=true , address=45.91.116.14:4190 , piece_id=KSMDOFLHHFMNWWUOY7GIJXNGQYDUUUMRIODAKTMUMKCRMMUPBAZA, seg_pos={0 0}, upload=517022]: Long tail cancellation due to optimal threshold Dec 30 05:16:23.313511207 (589h47m23.184594252s) [noise=true , address=77.90.31.14:4663 , piece_id=QB6J2XSJIJKST3TMID4FUTAL4RRCTC3GNYAVLKVAUZ76ZJ627THQ, seg_pos={0 0}, upload=517022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 30 05:16:23.313519192 (589h47m23.184602237s) [noise=true , address=77.90.31.14:4663 , piece_id=QB6J2XSJIJKST3TMID4FUTAL4RRCTC3GNYAVLKVAUZ76ZJ627THQ, seg_pos={0 0}, upload=517022]: Long tail cancellation due to optimal threshold Dec 30 05:16:23.313637757 (589h47m23.184720802s) [seg_pos={0 0}, upload=517022]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:16:23.313915486 (589h47m23.184998531s) [upload=517022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:16:23.313938711 (589h47m23.185021746s) [upload=517022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:16:23.603518878 (589h47m23.474601923s) [upload=517022]: Done uploading segments... Dec 30 05:16:23.603547282 (589h47m23.474630327s) [upload=517022]: Upload finished. err: <nil> Duration: 30.002751246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.....X.....X..............................] duration: 27.174405978s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:16:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 27.174433871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X.............X...........X............X.X....X........X.] duration: 3.563508051s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:16:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563542709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.XXX...X.X] duration: 30.003335755s trace: https://jaeger.dev.storj.io/trace/59bfb22bec6a8d5f FAIL (2025-12-30 05:16:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: 3UTVB5F5ZXKMFTMDN34X45RZ4WY5DTZW63PLVBE6OKWB7AXKAVFQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: JBWNSL3CJGHTT5ZFSM7A3RKUNFC5QWATDPPVBIPWGBDWI3OLE3DA) context deadline exceeded ======LOGS====== 0 05:15:50.496886076 (589h46m50.367969121s) [upload=517013]: Starting upload Dec 30 05:15:50.496994713 (589h46m50.368077758s) [upload=517013]: Done starting upload Dec 30 05:15:50.497109862 (589h46m50.368192907s) [upload=517013]: Uploading segments... Dec 30 05:15:50.497420794 (589h46m50.368503919s) [upload=517013]: Got next segment. Inline: false Dec 30 05:15:50.497487531 (589h46m50.368570586s) [upload=517013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:15:50.497496408 (589h46m50.368579453s) [upload=517013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:15:50.741447520 (589h46m50.612530555s) [seg_pos={0 0}, upload=517013]: Begin upload segment... Dec 30 05:15:50.741496513 (589h46m50.612579548s) [seg_pos={0 0}, upload=517013]: Done begin upload segment. Dec 30 05:15:50.741523214 (589h46m50.612606249s) [noise=true , address=45.91.116.5:4297 , piece_id=PFVOPKYNOLPOJZFCFXGBC455LF46MHGZABZOHDPFHB5V4NZYWKYQ, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.741630668 (589h46m50.612713704s) [noise=true , address=145.79.227.27:4689 , piece_id=ZJH757ODSENIFE2BCQV27ERY7XX46OTEBEGMGFNSZ466NISH6LUA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.742191877 (589h46m50.613274912s) [noise=true , address=145.79.227.15:4876 , piece_id=KBJ4KKTEM5GNUYJVQ4JUVES4YWU3TIC426UGVXQN3PHLIBHDZEYQ, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.742750240 (589h46m50.613833275s) [noise=true , address=77.90.31.14:4662 , piece_id=3UTVB5F5ZXKMFTMDN34X45RZ4WY5DTZW63PLVBE6OKWB7AXKAVFQ, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.743164820 (589h46m50.614247855s) [noise=true , address=145.79.227.2:4115 , piece_id=DB7RHSREHYHITHFOS6F2XAL3RA3T3K2V35PDHGKTX3IMMNEDN2GA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.743582325 (589h46m50.61466536s) [noise=true , address=145.79.227.50:4374 , piece_id=6DONJ7UFZ3IXCCURQ4Z46O6WSGZSIPHHHO3N5KHDJY2HB4BUTLUA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.744027473 (589h46m50.615110508s) [noise=true , address=145.79.227.19:4079 , piece_id=KEKWVN7N4LT55C2JTMJQO4MZB2L4LEAFYE6SNJSQYWOPQ65OFDVQ, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.745152825 (589h46m50.61623586s) [noise=true , address=145.79.227.9:4367 , piece_id=GNEFNZWMADPHKFPXI2PNS3FHTIRFUGFMGFH627RPDVXGJDKXD4LQ, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.745491941 (589h46m50.616574976s) [noise=true , address=145.79.227.20:4438 , piece_id=TAREJIKEFCDFXHGXJXVEDOIO3JQNGUMO2GPN6TQRTQPONDRQ2ALA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.746481296 (589h46m50.617564341s) [noise=true , address=45.91.116.10:4081 , piece_id=JBWNSL3CJGHTT5ZFSM7A3RKUNFC5QWATDPPVBIPWGBDWI3OLE3DA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.747146041 (589h46m50.618229076s) [noise=true , address=145.79.227.22:4509 , piece_id=RABEXB4M6FGC4VMNRYQEHDKN5F5RWMTW2JHRPVN7T2HKHZ7KZPKA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.747817259 (589h46m50.618900304s) [noise=true , address=45.91.116.12:4228 , piece_id=FW34HV2VMZ6B6XAMKLBDECOGQ4C3WUS2BEYPQY2ESM3JTKDAXCKA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.749706496 (589h46m50.620789541s) [noise=true , address=145.79.227.37:4567 , piece_id=IOS32ZSDL6JVGM2QLYSLOKDAQZP3VQMHEXJORIEE4IU3F7QDGSNQ, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.754207848 (589h46m50.625290893s) [noise=true , address=145.79.227.10:4402 , piece_id=4ZUBYQYZXBBLQDNZJIAHO6XXCZINWJDCDFP4D3BOQBCWCAALHWNA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.759261140 (589h46m50.630344186s) [noise=true , address=145.79.227.14:4839 , piece_id=6JSHE3UY6IPSCDAMPSNVCHF525QWBUNSWGM34STC3OMXIHPKSGLA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:50.762524564 (589h46m50.633607609s) [noise=true , address=45.91.116.4:4937 , piece_id=RPEE72Z2XFWHAV4M5YGROP6Q5PBQWQ3NDRLBL757CV7FA7KTIRCA, seg_pos={0 0}, upload=517013]: Uploading piece... Dec 30 05:15:52.359495615 (589h46m52.23057866s) [upload=517013]: Next returned nil segment Dec 30 05:15:52.359514700 (589h46m52.230597735s) [upload=517013]: Waiting for error group managing segments... Dec 30 05:15:52.733291362 (589h46m52.604374407s) [noise=true , address=145.79.227.19:4079 , piece_id=KEKWVN7N4LT55C2JTMJQO4MZB2L4LEAFYE6SNJSQYWOPQ65OFDVQ, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.738809429 (589h46m52.609892464s) [noise=true , address=145.79.227.20:4438 , piece_id=TAREJIKEFCDFXHGXJXVEDOIO3JQNGUMO2GPN6TQRTQPONDRQ2ALA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.746043544 (589h46m52.617126589s) [noise=true , address=145.79.227.27:4689 , piece_id=ZJH757ODSENIFE2BCQV27ERY7XX46OTEBEGMGFNSZ466NISH6LUA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.748242030 (589h46m52.619325066s) [noise=true , address=145.79.227.15:4876 , piece_id=KBJ4KKTEM5GNUYJVQ4JUVES4YWU3TIC426UGVXQN3PHLIBHDZEYQ, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.751032544 (589h46m52.622115579s) [noise=true , address=145.79.227.2:4115 , piece_id=DB7RHSREHYHITHFOS6F2XAL3RA3T3K2V35PDHGKTX3IMMNEDN2GA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.752952339 (589h46m52.624035384s) [noise=true , address=145.79.227.9:4367 , piece_id=GNEFNZWMADPHKFPXI2PNS3FHTIRFUGFMGFH627RPDVXGJDKXD4LQ, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.753245408 (589h46m52.624328453s) [noise=true , address=145.79.227.37:4567 , piece_id=IOS32ZSDL6JVGM2QLYSLOKDAQZP3VQMHEXJORIEE4IU3F7QDGSNQ, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.753685125 (589h46m52.62476817s) [noise=true , address=145.79.227.14:4839 , piece_id=6JSHE3UY6IPSCDAMPSNVCHF525QWBUNSWGM34STC3OMXIHPKSGLA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.754822060 (589h46m52.625905105s) [noise=true , address=145.79.227.10:4402 , piece_id=4ZUBYQYZXBBLQDNZJIAHO6XXCZINWJDCDFP4D3BOQBCWCAALHWNA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.762425879 (589h46m52.633508924s) [noise=true , address=145.79.227.22:4509 , piece_id=RABEXB4M6FGC4VMNRYQEHDKN5F5RWMTW2JHRPVN7T2HKHZ7KZPKA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:52.762779001 (589h46m52.633862036s) [noise=true , address=145.79.227.50:4374 , piece_id=6DONJ7UFZ3IXCCURQ4Z46O6WSGZSIPHHHO3N5KHDJY2HB4BUTLUA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:53.021875753 (589h46m52.892958798s) [noise=true , address=77.90.31.14:4662 , piece_id=3UTVB5F5ZXKMFTMDN34X45RZ4WY5DTZW63PLVBE6OKWB7AXKAVFQ, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:53.037459177 (589h46m52.908542222s) [noise=true , address=45.91.116.10:4081 , piece_id=JBWNSL3CJGHTT5ZFSM7A3RKUNFC5QWATDPPVBIPWGBDWI3OLE3DA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: <nil> Dec 30 05:15:53.037484204 (589h46m52.908567239s) [seg_pos={0 0}, upload=517013]: Segment reached optimal threshold of 13 pieces. Dec 30 05:15:53.037865299 (589h46m52.908948334s) [noise=true , address=45.91.116.5:4297 , piece_id=PFVOPKYNOLPOJZFCFXGBC455LF46MHGZABZOHDPFHB5V4NZYWKYQ, seg_pos={0 0}, upload=517013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:34148->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:34148->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34148->45.91.116.5:4297: use of closed network connection Dec 30 05:15:53.037880930 (589h46m52.908963965s) [noise=true , address=45.91.116.5:4297 , piece_id=PFVOPKYNOLPOJZFCFXGBC455LF46MHGZABZOHDPFHB5V4NZYWKYQ, seg_pos={0 0}, upload=517013]: Long tail cancellation due to optimal threshold Dec 30 05:15:53.038086581 (589h46m52.909169626s) [noise=true , address=45.91.116.12:4228 , piece_id=FW34HV2VMZ6B6XAMKLBDECOGQ4C3WUS2BEYPQY2ESM3JTKDAXCKA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 30 05:15:53.038103202 (589h46m52.909186248s) [noise=true , address=45.91.116.12:4228 , piece_id=FW34HV2VMZ6B6XAMKLBDECOGQ4C3WUS2BEYPQY2ESM3JTKDAXCKA, seg_pos={0 0}, upload=517013]: Long tail cancellation due to optimal threshold Dec 30 05:15:53.038137808 (589h46m52.909220843s) [noise=true , address=45.91.116.4:4937 , piece_id=RPEE72Z2XFWHAV4M5YGROP6Q5PBQWQ3NDRLBL757CV7FA7KTIRCA, seg_pos={0 0}, upload=517013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 05:15:53.038145493 (589h46m52.909228528s) [noise=true , address=45.91.116.4:4937 , piece_id=RPEE72Z2XFWHAV4M5YGROP6Q5PBQWQ3NDRLBL757CV7FA7KTIRCA, seg_pos={0 0}, upload=517013]: Long tail cancellation due to optimal threshold Dec 30 05:15:53.038161804 (589h46m52.909244839s) [seg_pos={0 0}, upload=517013]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:15:53.038330886 (589h46m52.909413921s) [upload=517013]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:15:53.038338010 (589h46m52.909421055s) [upload=517013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:15:53.331484919 (589h46m53.202567954s) [upload=517013]: Done uploading segments... Dec 30 05:15:53.331506881 (589h46m53.202589916s) [upload=517013]: Upload finished. err: <nil> Duration: 30.003361674s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X.............X...........X............X.X....X........X] duration: 5.001278749s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:15:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001291833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.....X.....X.............................] duration: 27.745535608s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:15:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 27.74555787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.XXX...X.] duration: 15.867507488s trace: https://jaeger.dev.storj.io/trace/750ecd34c11c9f71 FAIL (2025-12-30 05:15:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: 7HU5EOSGPJFCL7YL3UJMAMPXWA42LBPSVXE5QPX7DE5EGMVXNVKA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: U3DK6D6QXGBAKVPMYG4QR7HDBY6JYT4U2ZQIMZYIK2C2VTXI2SHQ) context deadline exceeded ======LOGS====== Dec 30 05:14:48.036336202 (589h45m47.907419237s) [upload=516998]: Starting upload Dec 30 05:14:48.036408770 (589h45m47.907491835s) [upload=516998]: Done starting upload Dec 30 05:14:48.036545430 (589h45m47.907628485s) [upload=516998]: Uploading segments... Dec 30 05:14:48.036611787 (589h45m47.907694832s) [upload=516998]: Got next segment. Inline: false Dec 30 05:14:48.036650160 (589h45m47.907733225s) [upload=516998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:14:48.036656582 (589h45m47.907739637s) [upload=516998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:14:48.278400762 (589h45m48.149483797s) [seg_pos={0 0}, upload=516998]: Begin upload segment... Dec 30 05:14:48.278457360 (589h45m48.149540405s) [seg_pos={0 0}, upload=516998]: Done begin upload segment. Dec 30 05:14:48.278480293 (589h45m48.149563338s) [noise=true , address=77.90.31.14:4661 , piece_id=7HU5EOSGPJFCL7YL3UJMAMPXWA42LBPSVXE5QPX7DE5EGMVXNVKA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.278508316 (589h45m48.149591351s) [noise=true , address=145.79.227.52:4446 , piece_id=ELJKMD4EXSJEVMKWV7KJDWIISZGQWFNJELIJWIRZR7ORMBGPJ4MQ, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.278628626 (589h45m48.149711661s) [noise=true , address=145.79.227.23:4547 , piece_id=FUYL7LQW2DWBVOEA3P4VW2BNC5KJUOEM3FAVWIKYL2GNG2KBF5LA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.279042864 (589h45m48.150125909s) [noise=true , address=145.79.227.19:4079 , piece_id=PU5BKOPLHZ4HNAAM2I6XIJT5TXESRMRUGH73E6P26V74MLW7TMGA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.279928481 (589h45m48.151011526s) [noise=true , address=145.79.227.28:4726 , piece_id=NMJ2OFAZ2CP7ISA4S3IBRS5MRROM77QVH4USRRGSXQ5O4OHUYOMQ, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.280433292 (589h45m48.151516337s) [noise=true , address=77.90.31.16:4736 , piece_id=U3DK6D6QXGBAKVPMYG4QR7HDBY6JYT4U2ZQIMZYIK2C2VTXI2SHQ, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.281010541 (589h45m48.152093586s) [noise=true , address=145.79.227.15:4873 , piece_id=3D2NFPIL7Y6DL4UYMYCBYSGGVFFAG7XM2QJNJGXTHI5XR33Z6L4Q, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.281523338 (589h45m48.152606383s) [noise=true , address=77.90.31.13:4627 , piece_id=QEZIQMMQDH54PVKXB42W7ALFZPPA7G2IQ522P4CRPD6OVNIAKSWA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.282041184 (589h45m48.153124229s) [noise=true , address=145.79.227.21:4473 , piece_id=GZAO4UAEFJF7WYYQ3BZ25BJYD77F32LH7BUAOBNJJ2ZY3CZZWY7A, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.282655794 (589h45m48.153738839s) [noise=true , address=145.79.227.7:4294 , piece_id=7H3EMPQVGI74D23OTWUQF4ZOPZVFOU7HGLJ4EE2HODATQYAA4Z3A, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.287791273 (589h45m48.158874308s) [noise=true , address=45.91.116.13:4153 , piece_id=K56ATCANECCS3FGR55Z4U7ARBW7TL7NLU2K3QVLRL2IL32IAX6GA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.294913945 (589h45m48.16599698s) [noise=true , address=145.79.227.13:4802 , piece_id=EOM3L5N3ECC4SIIQH5NOQC3LQWT2WH2A6ZIP2EMTUSBRLENVPEIQ, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.297475423 (589h45m48.168558468s) [noise=true , address=145.79.227.38:4977 , piece_id=GFDS5PHIAU3R3ICQNLSGHMHBM3UAYEEUK37ZFQI7UGPLAWDS6MGA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.299075309 (589h45m48.170158354s) [noise=true , address=77.90.31.21:4877 , piece_id=F3UYZVQSTVKLRWYKFJYGE7BDAVA3P422LVHQCKE3FJS2GLPJDLDA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.300690886 (589h45m48.171773931s) [noise=true , address=145.79.227.24:4582 , piece_id=DBVR6LEB7DAYEKY55NODYLTH5RLHIT7XVX4527WWTPDQERPUNDKA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.303525723 (589h45m48.174608758s) [noise=true , address=145.79.227.11:4730 , piece_id=OQMLPZOSXF4IXQQMXI2LBF3QAAQCHG3RRU2T5ADCG4FM2XGFZWKA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:49.979758191 (589h45m49.850841237s) [upload=516998]: Next returned nil segment Dec 30 05:14:49.979784612 (589h45m49.850867687s) [upload=516998]: Waiting for error group managing segments... Dec 30 05:14:50.265236778 (589h45m50.136319823s) [noise=true , address=145.79.227.7:4294 , piece_id=7H3EMPQVGI74D23OTWUQF4ZOPZVFOU7HGLJ4EE2HODATQYAA4Z3A, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.267969372 (589h45m50.139052407s) [noise=true , address=145.79.227.52:4446 , piece_id=ELJKMD4EXSJEVMKWV7KJDWIISZGQWFNJELIJWIRZR7ORMBGPJ4MQ, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.278946053 (589h45m50.150029088s) [noise=true , address=145.79.227.19:4079 , piece_id=PU5BKOPLHZ4HNAAM2I6XIJT5TXESRMRUGH73E6P26V74MLW7TMGA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.280988463 (589h45m50.152071508s) [noise=true , address=145.79.227.13:4802 , piece_id=EOM3L5N3ECC4SIIQH5NOQC3LQWT2WH2A6ZIP2EMTUSBRLENVPEIQ, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.286474168 (589h45m50.157557213s) [noise=true , address=145.79.227.23:4547 , piece_id=FUYL7LQW2DWBVOEA3P4VW2BNC5KJUOEM3FAVWIKYL2GNG2KBF5LA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.292879094 (589h45m50.163962129s) [noise=true , address=145.79.227.21:4473 , piece_id=GZAO4UAEFJF7WYYQ3BZ25BJYD77F32LH7BUAOBNJJ2ZY3CZZWY7A, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.304224738 (589h45m50.175307773s) [noise=true , address=145.79.227.24:4582 , piece_id=DBVR6LEB7DAYEKY55NODYLTH5RLHIT7XVX4527WWTPDQERPUNDKA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.304468622 (589h45m50.175551667s) [noise=true , address=145.79.227.38:4977 , piece_id=GFDS5PHIAU3R3ICQNLSGHMHBM3UAYEEUK37ZFQI7UGPLAWDS6MGA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.305888937 (589h45m50.176971982s) [noise=true , address=145.79.227.11:4730 , piece_id=OQMLPZOSXF4IXQQMXI2LBF3QAAQCHG3RRU2T5ADCG4FM2XGFZWKA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.310663779 (589h45m50.181746814s) [noise=true , address=145.79.227.15:4873 , piece_id=3D2NFPIL7Y6DL4UYMYCBYSGGVFFAG7XM2QJNJGXTHI5XR33Z6L4Q, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.317315625 (589h45m50.18839866s) [noise=true , address=145.79.227.28:4726 , piece_id=NMJ2OFAZ2CP7ISA4S3IBRS5MRROM77QVH4USRRGSXQ5O4OHUYOMQ, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.540571140 (589h45m50.411654205s) [noise=true , address=77.90.31.16:4736 , piece_id=U3DK6D6QXGBAKVPMYG4QR7HDBY6JYT4U2ZQIMZYIK2C2VTXI2SHQ, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.541106078 (589h45m50.412189123s) [noise=true , address=77.90.31.14:4661 , piece_id=7HU5EOSGPJFCL7YL3UJMAMPXWA42LBPSVXE5QPX7DE5EGMVXNVKA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.541116989 (589h45m50.412200034s) [seg_pos={0 0}, upload=516998]: Segment reached optimal threshold of 13 pieces. Dec 30 05:14:50.541398695 (589h45m50.41248174s) [noise=true , address=77.90.31.13:4627 , piece_id=QEZIQMMQDH54PVKXB42W7ALFZPPA7G2IQ522P4CRPD6OVNIAKSWA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 30 05:14:50.541407072 (589h45m50.412490117s) [noise=true , address=77.90.31.13:4627 , piece_id=QEZIQMMQDH54PVKXB42W7ALFZPPA7G2IQ522P4CRPD6OVNIAKSWA, seg_pos={0 0}, upload=516998]: Long tail cancellation due to optimal threshold Dec 30 05:14:50.541431427 (589h45m50.412514472s) [noise=true , address=77.90.31.21:4877 , piece_id=F3UYZVQSTVKLRWYKFJYGE7BDAVA3P422LVHQCKE3FJS2GLPJDLDA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 30 05:14:50.541462416 (589h45m50.412545451s) [noise=true , address=77.90.31.21:4877 , piece_id=F3UYZVQSTVKLRWYKFJYGE7BDAVA3P422LVHQCKE3FJS2GLPJDLDA, seg_pos={0 0}, upload=516998]: Long tail cancellation due to optimal threshold Dec 30 05:14:50.541539293 (589h45m50.412622348s) [noise=true , address=45.91.116.13:4153 , piece_id=K56ATCANECCS3FGR55Z4U7ARBW7TL7NLU2K3QVLRL2IL32IAX6GA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 05:14:50.541546768 (589h45m50.412629813s) [noise=true , address=45.91.116.13:4153 , piece_id=K56ATCANECCS3FGR55Z4U7ARBW7TL7NLU2K3QVLRL2IL32IAX6GA, seg_pos={0 0}, upload=516998]: Long tail cancellation due to optimal threshold Dec 30 05:14:50.541566915 (589h45m50.41264995s) [seg_pos={0 0}, upload=516998]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:14:50.541651317 (589h45m50.412734362s) [upload=516998]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:14:50.541657769 (589h45m50.412740804s) [upload=516998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:14:50.838843357 (589h45m50.709926392s) [upload=516998]: Done uploading segments... Dec 30 05:14:50.838878775 (589h45m50.70996183s) [upload=516998]: Upload finished. err: <nil> Duration: 15.867535651s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X.............X...........X............X.X....X........] duration: 3.814965058s trace: https://jaeger.dev.storj.io/trace/6b078ceaf59e0eb7 FAIL (2025-12-30 05:15:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.814996919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.XXX...X] duration: 30.004402531s trace: https://jaeger.dev.storj.io/trace/750ecd34c11c9f71 FAIL (2025-12-30 05:15:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: 7HU5EOSGPJFCL7YL3UJMAMPXWA42LBPSVXE5QPX7DE5EGMVXNVKA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: U3DK6D6QXGBAKVPMYG4QR7HDBY6JYT4U2ZQIMZYIK2C2VTXI2SHQ) context deadline exceeded ======LOGS====== Dec 30 05:14:48.036336202 (589h45m47.907419237s) [upload=516998]: Starting upload Dec 30 05:14:48.036408770 (589h45m47.907491835s) [upload=516998]: Done starting upload Dec 30 05:14:48.036545430 (589h45m47.907628485s) [upload=516998]: Uploading segments... Dec 30 05:14:48.036611787 (589h45m47.907694832s) [upload=516998]: Got next segment. Inline: false Dec 30 05:14:48.036650160 (589h45m47.907733225s) [upload=516998]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:14:48.036656582 (589h45m47.907739637s) [upload=516998]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:14:48.278400762 (589h45m48.149483797s) [seg_pos={0 0}, upload=516998]: Begin upload segment... Dec 30 05:14:48.278457360 (589h45m48.149540405s) [seg_pos={0 0}, upload=516998]: Done begin upload segment. Dec 30 05:14:48.278480293 (589h45m48.149563338s) [noise=true , address=77.90.31.14:4661 , piece_id=7HU5EOSGPJFCL7YL3UJMAMPXWA42LBPSVXE5QPX7DE5EGMVXNVKA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.278508316 (589h45m48.149591351s) [noise=true , address=145.79.227.52:4446 , piece_id=ELJKMD4EXSJEVMKWV7KJDWIISZGQWFNJELIJWIRZR7ORMBGPJ4MQ, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.278628626 (589h45m48.149711661s) [noise=true , address=145.79.227.23:4547 , piece_id=FUYL7LQW2DWBVOEA3P4VW2BNC5KJUOEM3FAVWIKYL2GNG2KBF5LA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.279042864 (589h45m48.150125909s) [noise=true , address=145.79.227.19:4079 , piece_id=PU5BKOPLHZ4HNAAM2I6XIJT5TXESRMRUGH73E6P26V74MLW7TMGA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.279928481 (589h45m48.151011526s) [noise=true , address=145.79.227.28:4726 , piece_id=NMJ2OFAZ2CP7ISA4S3IBRS5MRROM77QVH4USRRGSXQ5O4OHUYOMQ, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.280433292 (589h45m48.151516337s) [noise=true , address=77.90.31.16:4736 , piece_id=U3DK6D6QXGBAKVPMYG4QR7HDBY6JYT4U2ZQIMZYIK2C2VTXI2SHQ, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.281010541 (589h45m48.152093586s) [noise=true , address=145.79.227.15:4873 , piece_id=3D2NFPIL7Y6DL4UYMYCBYSGGVFFAG7XM2QJNJGXTHI5XR33Z6L4Q, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.281523338 (589h45m48.152606383s) [noise=true , address=77.90.31.13:4627 , piece_id=QEZIQMMQDH54PVKXB42W7ALFZPPA7G2IQ522P4CRPD6OVNIAKSWA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.282041184 (589h45m48.153124229s) [noise=true , address=145.79.227.21:4473 , piece_id=GZAO4UAEFJF7WYYQ3BZ25BJYD77F32LH7BUAOBNJJ2ZY3CZZWY7A, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.282655794 (589h45m48.153738839s) [noise=true , address=145.79.227.7:4294 , piece_id=7H3EMPQVGI74D23OTWUQF4ZOPZVFOU7HGLJ4EE2HODATQYAA4Z3A, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.287791273 (589h45m48.158874308s) [noise=true , address=45.91.116.13:4153 , piece_id=K56ATCANECCS3FGR55Z4U7ARBW7TL7NLU2K3QVLRL2IL32IAX6GA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.294913945 (589h45m48.16599698s) [noise=true , address=145.79.227.13:4802 , piece_id=EOM3L5N3ECC4SIIQH5NOQC3LQWT2WH2A6ZIP2EMTUSBRLENVPEIQ, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.297475423 (589h45m48.168558468s) [noise=true , address=145.79.227.38:4977 , piece_id=GFDS5PHIAU3R3ICQNLSGHMHBM3UAYEEUK37ZFQI7UGPLAWDS6MGA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.299075309 (589h45m48.170158354s) [noise=true , address=77.90.31.21:4877 , piece_id=F3UYZVQSTVKLRWYKFJYGE7BDAVA3P422LVHQCKE3FJS2GLPJDLDA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.300690886 (589h45m48.171773931s) [noise=true , address=145.79.227.24:4582 , piece_id=DBVR6LEB7DAYEKY55NODYLTH5RLHIT7XVX4527WWTPDQERPUNDKA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:48.303525723 (589h45m48.174608758s) [noise=true , address=145.79.227.11:4730 , piece_id=OQMLPZOSXF4IXQQMXI2LBF3QAAQCHG3RRU2T5ADCG4FM2XGFZWKA, seg_pos={0 0}, upload=516998]: Uploading piece... Dec 30 05:14:49.979758191 (589h45m49.850841237s) [upload=516998]: Next returned nil segment Dec 30 05:14:49.979784612 (589h45m49.850867687s) [upload=516998]: Waiting for error group managing segments... Dec 30 05:14:50.265236778 (589h45m50.136319823s) [noise=true , address=145.79.227.7:4294 , piece_id=7H3EMPQVGI74D23OTWUQF4ZOPZVFOU7HGLJ4EE2HODATQYAA4Z3A, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.267969372 (589h45m50.139052407s) [noise=true , address=145.79.227.52:4446 , piece_id=ELJKMD4EXSJEVMKWV7KJDWIISZGQWFNJELIJWIRZR7ORMBGPJ4MQ, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.278946053 (589h45m50.150029088s) [noise=true , address=145.79.227.19:4079 , piece_id=PU5BKOPLHZ4HNAAM2I6XIJT5TXESRMRUGH73E6P26V74MLW7TMGA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.280988463 (589h45m50.152071508s) [noise=true , address=145.79.227.13:4802 , piece_id=EOM3L5N3ECC4SIIQH5NOQC3LQWT2WH2A6ZIP2EMTUSBRLENVPEIQ, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.286474168 (589h45m50.157557213s) [noise=true , address=145.79.227.23:4547 , piece_id=FUYL7LQW2DWBVOEA3P4VW2BNC5KJUOEM3FAVWIKYL2GNG2KBF5LA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.292879094 (589h45m50.163962129s) [noise=true , address=145.79.227.21:4473 , piece_id=GZAO4UAEFJF7WYYQ3BZ25BJYD77F32LH7BUAOBNJJ2ZY3CZZWY7A, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.304224738 (589h45m50.175307773s) [noise=true , address=145.79.227.24:4582 , piece_id=DBVR6LEB7DAYEKY55NODYLTH5RLHIT7XVX4527WWTPDQERPUNDKA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.304468622 (589h45m50.175551667s) [noise=true , address=145.79.227.38:4977 , piece_id=GFDS5PHIAU3R3ICQNLSGHMHBM3UAYEEUK37ZFQI7UGPLAWDS6MGA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.305888937 (589h45m50.176971982s) [noise=true , address=145.79.227.11:4730 , piece_id=OQMLPZOSXF4IXQQMXI2LBF3QAAQCHG3RRU2T5ADCG4FM2XGFZWKA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.310663779 (589h45m50.181746814s) [noise=true , address=145.79.227.15:4873 , piece_id=3D2NFPIL7Y6DL4UYMYCBYSGGVFFAG7XM2QJNJGXTHI5XR33Z6L4Q, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.317315625 (589h45m50.18839866s) [noise=true , address=145.79.227.28:4726 , piece_id=NMJ2OFAZ2CP7ISA4S3IBRS5MRROM77QVH4USRRGSXQ5O4OHUYOMQ, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.540571140 (589h45m50.411654205s) [noise=true , address=77.90.31.16:4736 , piece_id=U3DK6D6QXGBAKVPMYG4QR7HDBY6JYT4U2ZQIMZYIK2C2VTXI2SHQ, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.541106078 (589h45m50.412189123s) [noise=true , address=77.90.31.14:4661 , piece_id=7HU5EOSGPJFCL7YL3UJMAMPXWA42LBPSVXE5QPX7DE5EGMVXNVKA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: <nil> Dec 30 05:14:50.541116989 (589h45m50.412200034s) [seg_pos={0 0}, upload=516998]: Segment reached optimal threshold of 13 pieces. Dec 30 05:14:50.541398695 (589h45m50.41248174s) [noise=true , address=77.90.31.13:4627 , piece_id=QEZIQMMQDH54PVKXB42W7ALFZPPA7G2IQ522P4CRPD6OVNIAKSWA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 30 05:14:50.541407072 (589h45m50.412490117s) [noise=true , address=77.90.31.13:4627 , piece_id=QEZIQMMQDH54PVKXB42W7ALFZPPA7G2IQ522P4CRPD6OVNIAKSWA, seg_pos={0 0}, upload=516998]: Long tail cancellation due to optimal threshold Dec 30 05:14:50.541431427 (589h45m50.412514472s) [noise=true , address=77.90.31.21:4877 , piece_id=F3UYZVQSTVKLRWYKFJYGE7BDAVA3P422LVHQCKE3FJS2GLPJDLDA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 30 05:14:50.541462416 (589h45m50.412545451s) [noise=true , address=77.90.31.21:4877 , piece_id=F3UYZVQSTVKLRWYKFJYGE7BDAVA3P422LVHQCKE3FJS2GLPJDLDA, seg_pos={0 0}, upload=516998]: Long tail cancellation due to optimal threshold Dec 30 05:14:50.541539293 (589h45m50.412622348s) [noise=true , address=45.91.116.13:4153 , piece_id=K56ATCANECCS3FGR55Z4U7ARBW7TL7NLU2K3QVLRL2IL32IAX6GA, seg_pos={0 0}, upload=516998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 05:14:50.541546768 (589h45m50.412629813s) [noise=true , address=45.91.116.13:4153 , piece_id=K56ATCANECCS3FGR55Z4U7ARBW7TL7NLU2K3QVLRL2IL32IAX6GA, seg_pos={0 0}, upload=516998]: Long tail cancellation due to optimal threshold Dec 30 05:14:50.541566915 (589h45m50.41264995s) [seg_pos={0 0}, upload=516998]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:14:50.541651317 (589h45m50.412734362s) [upload=516998]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:14:50.541657769 (589h45m50.412740804s) [upload=516998]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:14:50.838843357 (589h45m50.709926392s) [upload=516998]: Done uploading segments... Dec 30 05:14:50.838878775 (589h45m50.70996183s) [upload=516998]: Upload finished. err: <nil> Duration: 30.004429882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.....X.....X............................] duration: 31.761734417s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:15:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 31.76176169s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X.............X...........X............X.X....X.......] duration: 3.575880553s trace: https://jaeger.dev.storj.io/trace/6b078ceaf59e0eb7 FAIL (2025-12-30 05:14:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.575904589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.....X.....X...........................] duration: 47.68944717s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:14:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 47.689470986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.XXX...] duration: 5.916863441s trace: https://jaeger.dev.storj.io/trace/322b49be4d65d745 FAIL (2025-12-30 05:13:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q) context deadline exceeded ======LOGS====== Dec 30 05:10:58.676089142 (589h41m58.547172187s) [upload=516942]: Starting upload Dec 30 05:10:58.676199973 (589h41m58.547283018s) [upload=516942]: Done starting upload Dec 30 05:10:58.676333828 (589h41m58.547416873s) [upload=516942]: Uploading segments... Dec 30 05:10:58.676508070 (589h41m58.547591115s) [upload=516942]: Got next segment. Inline: false Dec 30 05:10:58.676575919 (589h41m58.547658964s) [upload=516942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:10:58.676588103 (589h41m58.547671148s) [upload=516942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:10:58.917398906 (589h41m58.788481941s) [seg_pos={0 0}, upload=516942]: Begin upload segment... Dec 30 05:10:58.917455272 (589h41m58.788538307s) [seg_pos={0 0}, upload=516942]: Done begin upload segment. Dec 30 05:10:58.917481302 (589h41m58.788564337s) [noise=true , address=77.90.31.12:4590 , piece_id=6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.917569400 (589h41m58.788652435s) [noise=true , address=145.79.227.45:4230 , piece_id=RFOYZLJ7BSB6PP3CMQ2SQHKKAUTFSW2AI6737RMDKE2HHACZZVXA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918247341 (589h41m58.789330376s) [noise=true , address=145.79.227.17:4971 , piece_id=DNAVBWG73I6UF76U36YQTMMUJ7J2UP45QGX2SDFVANVNUME7RJZA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918687499 (589h41m58.789770534s) [noise=true , address=45.91.116.14:4191 , piece_id=QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918975738 (589h41m58.790058783s) [noise=true , address=145.79.227.11:4732 , piece_id=R3QU3O6NDVR5YBEPHI7UMJ63IKAPUJWJY2S2ZE47N5CQCRXFMFFA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.919416117 (589h41m58.790499152s) [noise=true , address=145.79.227.20:4438 , piece_id=I4NRHBOM67LOKR6WO3LNANBBHH6UJVJFH3LNQKJTMA74TCWDYA4Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.919725155 (589h41m58.79080819s) [noise=true , address=45.91.116.11:4117 , piece_id=W7KKEY6AXB7PMVYMDOZNPACFJP5KDSTZGVLT4RQU7H54H2YGF53Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920082305 (589h41m58.79116535s) [noise=true , address=145.79.227.14:4838 , piece_id=NXIMNSYGILEY4V7IXD4RWZW4X7KBF3VKG7OA6ULOEKCVP5VUNNWA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920476976 (589h41m58.791560011s) [noise=true , address=145.79.227.10:4401 , piece_id=52DK6S6UHVTTBNJQDZNJCQCZG4VJCAGU53FRNEZSHEXVGHDEP3WA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920931001 (589h41m58.792014036s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.922889700 (589h41m58.793972745s) [noise=true , address=145.79.227.6:4258 , piece_id=BQVTINULZFBEO77AFDDZZZINEWVCIKZ4JY5B7LHKBM7EM2VI2I5Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.924179015 (589h41m58.79526205s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.925304348 (589h41m58.796387393s) [noise=true , address=145.79.227.9:4366 , piece_id=SGV3HP3J2RHHR2T6RSKPQH5Z3GGKFSENXVJWYZNQJ5XPRQVZPM4Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.926549098 (589h41m58.797632133s) [noise=true , address=145.79.227.4:4186 , piece_id=Y5SEXLAN6GV3SYD4S5MDLKTPTDTTUC6BIFJIK53DL4J2OZYONSGQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.927665103 (589h41m58.798748148s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.928804724 (589h41m58.799887759s) [noise=true , address=45.91.116.13:4155 , piece_id=L6NQ4ATJU3ORHBEQBMZODIAJG6L25T4RKNOQ5CUMTXLOGQYM67BA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:11:00.522237737 (589h42m0.393320782s) [upload=516942]: Next returned nil segment Dec 30 05:11:00.522253286 (589h42m0.393336331s) [upload=516942]: Waiting for error group managing segments... Dec 30 05:11:00.904517749 (589h42m0.775600785s) [noise=true , address=145.79.227.11:4732 , piece_id=R3QU3O6NDVR5YBEPHI7UMJ63IKAPUJWJY2S2ZE47N5CQCRXFMFFA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.908073510 (589h42m0.779156545s) [noise=true , address=145.79.227.14:4838 , piece_id=NXIMNSYGILEY4V7IXD4RWZW4X7KBF3VKG7OA6ULOEKCVP5VUNNWA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.909693365 (589h42m0.7807764s) [noise=true , address=145.79.227.45:4230 , piece_id=RFOYZLJ7BSB6PP3CMQ2SQHKKAUTFSW2AI6737RMDKE2HHACZZVXA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.910894281 (589h42m0.781977316s) [noise=true , address=145.79.227.6:4258 , piece_id=BQVTINULZFBEO77AFDDZZZINEWVCIKZ4JY5B7LHKBM7EM2VI2I5Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.928505957 (589h42m0.799589002s) [noise=true , address=145.79.227.10:4401 , piece_id=52DK6S6UHVTTBNJQDZNJCQCZG4VJCAGU53FRNEZSHEXVGHDEP3WA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.928990931 (589h42m0.800073966s) [noise=true , address=145.79.227.20:4438 , piece_id=I4NRHBOM67LOKR6WO3LNANBBHH6UJVJFH3LNQKJTMA74TCWDYA4Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.929344133 (589h42m0.800427178s) [noise=true , address=145.79.227.17:4971 , piece_id=DNAVBWG73I6UF76U36YQTMMUJ7J2UP45QGX2SDFVANVNUME7RJZA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.936862780 (589h42m0.807945835s) [noise=true , address=145.79.227.9:4366 , piece_id=SGV3HP3J2RHHR2T6RSKPQH5Z3GGKFSENXVJWYZNQJ5XPRQVZPM4Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.937960160 (589h42m0.809043205s) [noise=true , address=145.79.227.4:4186 , piece_id=Y5SEXLAN6GV3SYD4S5MDLKTPTDTTUC6BIFJIK53DL4J2OZYONSGQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.090721532 (589h42m0.961804577s) [noise=true , address=77.90.31.12:4590 , piece_id=6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.152347930 (589h42m1.023430975s) [noise=true , address=45.91.116.14:4191 , piece_id=QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.304445216 (589h42m1.175528261s) [noise=true , address=45.91.116.11:4117 , piece_id=W7KKEY6AXB7PMVYMDOZNPACFJP5KDSTZGVLT4RQU7H54H2YGF53Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.353503822 (589h42m1.224586867s) [noise=true , address=45.91.116.13:4155 , piece_id=L6NQ4ATJU3ORHBEQBMZODIAJG6L25T4RKNOQ5CUMTXLOGQYM67BA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.353533188 (589h42m1.224616233s) [seg_pos={0 0}, upload=516942]: Segment reached optimal threshold of 13 pieces. Dec 30 05:11:01.353777343 (589h42m1.224860388s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.353785237 (589h42m1.224868282s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354094436 (589h42m1.225177471s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.354103253 (589h42m1.225186288s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354415377 (589h42m1.225498422s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.354442529 (589h42m1.225525614s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354514726 (589h42m1.225597772s) [seg_pos={0 0}, upload=516942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:11:01.354671896 (589h42m1.225754941s) [upload=516942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:11:01.354679841 (589h42m1.225762886s) [upload=516942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:11:01.646260317 (589h42m1.517343382s) [upload=516942]: Done uploading segments... Dec 30 05:11:01.646402247 (589h42m1.517485292s) [upload=516942]: Upload finished. err: <nil> Duration: 5.916902596s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X.............X...........X............X.X....X......] duration: 3.37684675s trace: https://jaeger.dev.storj.io/trace/6b078ceaf59e0eb7 FAIL (2025-12-30 05:13:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.376868803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.....X.....X..........................] duration: 25.910182954s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:13:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 25.910234271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.XXX..] duration: 29.271715693s trace: https://jaeger.dev.storj.io/trace/322b49be4d65d745 FAIL (2025-12-30 05:13:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q) context deadline exceeded ======LOGS====== Dec 30 05:10:58.676089142 (589h41m58.547172187s) [upload=516942]: Starting upload Dec 30 05:10:58.676199973 (589h41m58.547283018s) [upload=516942]: Done starting upload Dec 30 05:10:58.676333828 (589h41m58.547416873s) [upload=516942]: Uploading segments... Dec 30 05:10:58.676508070 (589h41m58.547591115s) [upload=516942]: Got next segment. Inline: false Dec 30 05:10:58.676575919 (589h41m58.547658964s) [upload=516942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:10:58.676588103 (589h41m58.547671148s) [upload=516942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:10:58.917398906 (589h41m58.788481941s) [seg_pos={0 0}, upload=516942]: Begin upload segment... Dec 30 05:10:58.917455272 (589h41m58.788538307s) [seg_pos={0 0}, upload=516942]: Done begin upload segment. Dec 30 05:10:58.917481302 (589h41m58.788564337s) [noise=true , address=77.90.31.12:4590 , piece_id=6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.917569400 (589h41m58.788652435s) [noise=true , address=145.79.227.45:4230 , piece_id=RFOYZLJ7BSB6PP3CMQ2SQHKKAUTFSW2AI6737RMDKE2HHACZZVXA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918247341 (589h41m58.789330376s) [noise=true , address=145.79.227.17:4971 , piece_id=DNAVBWG73I6UF76U36YQTMMUJ7J2UP45QGX2SDFVANVNUME7RJZA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918687499 (589h41m58.789770534s) [noise=true , address=45.91.116.14:4191 , piece_id=QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918975738 (589h41m58.790058783s) [noise=true , address=145.79.227.11:4732 , piece_id=R3QU3O6NDVR5YBEPHI7UMJ63IKAPUJWJY2S2ZE47N5CQCRXFMFFA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.919416117 (589h41m58.790499152s) [noise=true , address=145.79.227.20:4438 , piece_id=I4NRHBOM67LOKR6WO3LNANBBHH6UJVJFH3LNQKJTMA74TCWDYA4Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.919725155 (589h41m58.79080819s) [noise=true , address=45.91.116.11:4117 , piece_id=W7KKEY6AXB7PMVYMDOZNPACFJP5KDSTZGVLT4RQU7H54H2YGF53Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920082305 (589h41m58.79116535s) [noise=true , address=145.79.227.14:4838 , piece_id=NXIMNSYGILEY4V7IXD4RWZW4X7KBF3VKG7OA6ULOEKCVP5VUNNWA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920476976 (589h41m58.791560011s) [noise=true , address=145.79.227.10:4401 , piece_id=52DK6S6UHVTTBNJQDZNJCQCZG4VJCAGU53FRNEZSHEXVGHDEP3WA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920931001 (589h41m58.792014036s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.922889700 (589h41m58.793972745s) [noise=true , address=145.79.227.6:4258 , piece_id=BQVTINULZFBEO77AFDDZZZINEWVCIKZ4JY5B7LHKBM7EM2VI2I5Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.924179015 (589h41m58.79526205s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.925304348 (589h41m58.796387393s) [noise=true , address=145.79.227.9:4366 , piece_id=SGV3HP3J2RHHR2T6RSKPQH5Z3GGKFSENXVJWYZNQJ5XPRQVZPM4Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.926549098 (589h41m58.797632133s) [noise=true , address=145.79.227.4:4186 , piece_id=Y5SEXLAN6GV3SYD4S5MDLKTPTDTTUC6BIFJIK53DL4J2OZYONSGQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.927665103 (589h41m58.798748148s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.928804724 (589h41m58.799887759s) [noise=true , address=45.91.116.13:4155 , piece_id=L6NQ4ATJU3ORHBEQBMZODIAJG6L25T4RKNOQ5CUMTXLOGQYM67BA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:11:00.522237737 (589h42m0.393320782s) [upload=516942]: Next returned nil segment Dec 30 05:11:00.522253286 (589h42m0.393336331s) [upload=516942]: Waiting for error group managing segments... Dec 30 05:11:00.904517749 (589h42m0.775600785s) [noise=true , address=145.79.227.11:4732 , piece_id=R3QU3O6NDVR5YBEPHI7UMJ63IKAPUJWJY2S2ZE47N5CQCRXFMFFA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.908073510 (589h42m0.779156545s) [noise=true , address=145.79.227.14:4838 , piece_id=NXIMNSYGILEY4V7IXD4RWZW4X7KBF3VKG7OA6ULOEKCVP5VUNNWA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.909693365 (589h42m0.7807764s) [noise=true , address=145.79.227.45:4230 , piece_id=RFOYZLJ7BSB6PP3CMQ2SQHKKAUTFSW2AI6737RMDKE2HHACZZVXA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.910894281 (589h42m0.781977316s) [noise=true , address=145.79.227.6:4258 , piece_id=BQVTINULZFBEO77AFDDZZZINEWVCIKZ4JY5B7LHKBM7EM2VI2I5Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.928505957 (589h42m0.799589002s) [noise=true , address=145.79.227.10:4401 , piece_id=52DK6S6UHVTTBNJQDZNJCQCZG4VJCAGU53FRNEZSHEXVGHDEP3WA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.928990931 (589h42m0.800073966s) [noise=true , address=145.79.227.20:4438 , piece_id=I4NRHBOM67LOKR6WO3LNANBBHH6UJVJFH3LNQKJTMA74TCWDYA4Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.929344133 (589h42m0.800427178s) [noise=true , address=145.79.227.17:4971 , piece_id=DNAVBWG73I6UF76U36YQTMMUJ7J2UP45QGX2SDFVANVNUME7RJZA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.936862780 (589h42m0.807945835s) [noise=true , address=145.79.227.9:4366 , piece_id=SGV3HP3J2RHHR2T6RSKPQH5Z3GGKFSENXVJWYZNQJ5XPRQVZPM4Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.937960160 (589h42m0.809043205s) [noise=true , address=145.79.227.4:4186 , piece_id=Y5SEXLAN6GV3SYD4S5MDLKTPTDTTUC6BIFJIK53DL4J2OZYONSGQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.090721532 (589h42m0.961804577s) [noise=true , address=77.90.31.12:4590 , piece_id=6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.152347930 (589h42m1.023430975s) [noise=true , address=45.91.116.14:4191 , piece_id=QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.304445216 (589h42m1.175528261s) [noise=true , address=45.91.116.11:4117 , piece_id=W7KKEY6AXB7PMVYMDOZNPACFJP5KDSTZGVLT4RQU7H54H2YGF53Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.353503822 (589h42m1.224586867s) [noise=true , address=45.91.116.13:4155 , piece_id=L6NQ4ATJU3ORHBEQBMZODIAJG6L25T4RKNOQ5CUMTXLOGQYM67BA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.353533188 (589h42m1.224616233s) [seg_pos={0 0}, upload=516942]: Segment reached optimal threshold of 13 pieces. Dec 30 05:11:01.353777343 (589h42m1.224860388s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.353785237 (589h42m1.224868282s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354094436 (589h42m1.225177471s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.354103253 (589h42m1.225186288s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354415377 (589h42m1.225498422s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.354442529 (589h42m1.225525614s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354514726 (589h42m1.225597772s) [seg_pos={0 0}, upload=516942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:11:01.354671896 (589h42m1.225754941s) [upload=516942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:11:01.354679841 (589h42m1.225762886s) [upload=516942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:11:01.646260317 (589h42m1.517343382s) [upload=516942]: Done uploading segments... Dec 30 05:11:01.646402247 (589h42m1.517485292s) [upload=516942]: Upload finished. err: <nil> Duration: 29.271748065s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........X.............X...........X............X.X....X.....] duration: 5.001119495s trace: https://jaeger.dev.storj.io/trace/6b078ceaf59e0eb7 FAIL (2025-12-30 05:13:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001160764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.XXX.] duration: 22.232904687s trace: https://jaeger.dev.storj.io/trace/322b49be4d65d745 FAIL (2025-12-30 05:12:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q) context deadline exceeded ======LOGS====== Dec 30 05:10:58.676089142 (589h41m58.547172187s) [upload=516942]: Starting upload Dec 30 05:10:58.676199973 (589h41m58.547283018s) [upload=516942]: Done starting upload Dec 30 05:10:58.676333828 (589h41m58.547416873s) [upload=516942]: Uploading segments... Dec 30 05:10:58.676508070 (589h41m58.547591115s) [upload=516942]: Got next segment. Inline: false Dec 30 05:10:58.676575919 (589h41m58.547658964s) [upload=516942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:10:58.676588103 (589h41m58.547671148s) [upload=516942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:10:58.917398906 (589h41m58.788481941s) [seg_pos={0 0}, upload=516942]: Begin upload segment... Dec 30 05:10:58.917455272 (589h41m58.788538307s) [seg_pos={0 0}, upload=516942]: Done begin upload segment. Dec 30 05:10:58.917481302 (589h41m58.788564337s) [noise=true , address=77.90.31.12:4590 , piece_id=6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.917569400 (589h41m58.788652435s) [noise=true , address=145.79.227.45:4230 , piece_id=RFOYZLJ7BSB6PP3CMQ2SQHKKAUTFSW2AI6737RMDKE2HHACZZVXA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918247341 (589h41m58.789330376s) [noise=true , address=145.79.227.17:4971 , piece_id=DNAVBWG73I6UF76U36YQTMMUJ7J2UP45QGX2SDFVANVNUME7RJZA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918687499 (589h41m58.789770534s) [noise=true , address=45.91.116.14:4191 , piece_id=QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918975738 (589h41m58.790058783s) [noise=true , address=145.79.227.11:4732 , piece_id=R3QU3O6NDVR5YBEPHI7UMJ63IKAPUJWJY2S2ZE47N5CQCRXFMFFA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.919416117 (589h41m58.790499152s) [noise=true , address=145.79.227.20:4438 , piece_id=I4NRHBOM67LOKR6WO3LNANBBHH6UJVJFH3LNQKJTMA74TCWDYA4Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.919725155 (589h41m58.79080819s) [noise=true , address=45.91.116.11:4117 , piece_id=W7KKEY6AXB7PMVYMDOZNPACFJP5KDSTZGVLT4RQU7H54H2YGF53Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920082305 (589h41m58.79116535s) [noise=true , address=145.79.227.14:4838 , piece_id=NXIMNSYGILEY4V7IXD4RWZW4X7KBF3VKG7OA6ULOEKCVP5VUNNWA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920476976 (589h41m58.791560011s) [noise=true , address=145.79.227.10:4401 , piece_id=52DK6S6UHVTTBNJQDZNJCQCZG4VJCAGU53FRNEZSHEXVGHDEP3WA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920931001 (589h41m58.792014036s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.922889700 (589h41m58.793972745s) [noise=true , address=145.79.227.6:4258 , piece_id=BQVTINULZFBEO77AFDDZZZINEWVCIKZ4JY5B7LHKBM7EM2VI2I5Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.924179015 (589h41m58.79526205s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.925304348 (589h41m58.796387393s) [noise=true , address=145.79.227.9:4366 , piece_id=SGV3HP3J2RHHR2T6RSKPQH5Z3GGKFSENXVJWYZNQJ5XPRQVZPM4Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.926549098 (589h41m58.797632133s) [noise=true , address=145.79.227.4:4186 , piece_id=Y5SEXLAN6GV3SYD4S5MDLKTPTDTTUC6BIFJIK53DL4J2OZYONSGQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.927665103 (589h41m58.798748148s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.928804724 (589h41m58.799887759s) [noise=true , address=45.91.116.13:4155 , piece_id=L6NQ4ATJU3ORHBEQBMZODIAJG6L25T4RKNOQ5CUMTXLOGQYM67BA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:11:00.522237737 (589h42m0.393320782s) [upload=516942]: Next returned nil segment Dec 30 05:11:00.522253286 (589h42m0.393336331s) [upload=516942]: Waiting for error group managing segments... Dec 30 05:11:00.904517749 (589h42m0.775600785s) [noise=true , address=145.79.227.11:4732 , piece_id=R3QU3O6NDVR5YBEPHI7UMJ63IKAPUJWJY2S2ZE47N5CQCRXFMFFA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.908073510 (589h42m0.779156545s) [noise=true , address=145.79.227.14:4838 , piece_id=NXIMNSYGILEY4V7IXD4RWZW4X7KBF3VKG7OA6ULOEKCVP5VUNNWA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.909693365 (589h42m0.7807764s) [noise=true , address=145.79.227.45:4230 , piece_id=RFOYZLJ7BSB6PP3CMQ2SQHKKAUTFSW2AI6737RMDKE2HHACZZVXA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.910894281 (589h42m0.781977316s) [noise=true , address=145.79.227.6:4258 , piece_id=BQVTINULZFBEO77AFDDZZZINEWVCIKZ4JY5B7LHKBM7EM2VI2I5Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.928505957 (589h42m0.799589002s) [noise=true , address=145.79.227.10:4401 , piece_id=52DK6S6UHVTTBNJQDZNJCQCZG4VJCAGU53FRNEZSHEXVGHDEP3WA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.928990931 (589h42m0.800073966s) [noise=true , address=145.79.227.20:4438 , piece_id=I4NRHBOM67LOKR6WO3LNANBBHH6UJVJFH3LNQKJTMA74TCWDYA4Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.929344133 (589h42m0.800427178s) [noise=true , address=145.79.227.17:4971 , piece_id=DNAVBWG73I6UF76U36YQTMMUJ7J2UP45QGX2SDFVANVNUME7RJZA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.936862780 (589h42m0.807945835s) [noise=true , address=145.79.227.9:4366 , piece_id=SGV3HP3J2RHHR2T6RSKPQH5Z3GGKFSENXVJWYZNQJ5XPRQVZPM4Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.937960160 (589h42m0.809043205s) [noise=true , address=145.79.227.4:4186 , piece_id=Y5SEXLAN6GV3SYD4S5MDLKTPTDTTUC6BIFJIK53DL4J2OZYONSGQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.090721532 (589h42m0.961804577s) [noise=true , address=77.90.31.12:4590 , piece_id=6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.152347930 (589h42m1.023430975s) [noise=true , address=45.91.116.14:4191 , piece_id=QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.304445216 (589h42m1.175528261s) [noise=true , address=45.91.116.11:4117 , piece_id=W7KKEY6AXB7PMVYMDOZNPACFJP5KDSTZGVLT4RQU7H54H2YGF53Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.353503822 (589h42m1.224586867s) [noise=true , address=45.91.116.13:4155 , piece_id=L6NQ4ATJU3ORHBEQBMZODIAJG6L25T4RKNOQ5CUMTXLOGQYM67BA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.353533188 (589h42m1.224616233s) [seg_pos={0 0}, upload=516942]: Segment reached optimal threshold of 13 pieces. Dec 30 05:11:01.353777343 (589h42m1.224860388s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.353785237 (589h42m1.224868282s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354094436 (589h42m1.225177471s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.354103253 (589h42m1.225186288s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354415377 (589h42m1.225498422s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.354442529 (589h42m1.225525614s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354514726 (589h42m1.225597772s) [seg_pos={0 0}, upload=516942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:11:01.354671896 (589h42m1.225754941s) [upload=516942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:11:01.354679841 (589h42m1.225762886s) [upload=516942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:11:01.646260317 (589h42m1.517343382s) [upload=516942]: Done uploading segments... Dec 30 05:11:01.646402247 (589h42m1.517485292s) [upload=516942]: Upload finished. err: <nil> Duration: 22.232937999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.....X.....X.........................] duration: 26.604282274s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:12:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 26.604297142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........X.............X...........X............X.X....X....] duration: 3.536297312s trace: https://jaeger.dev.storj.io/trace/6b078ceaf59e0eb7 FAIL (2025-12-30 05:11:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53632262s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.....X.....X........................] duration: 38.21389336s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:11:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 38.213913739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.XXX] duration: 30.004770719s trace: https://jaeger.dev.storj.io/trace/322b49be4d65d745 FAIL (2025-12-30 05:11:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q) context deadline exceeded ======LOGS====== Dec 30 05:10:58.676089142 (589h41m58.547172187s) [upload=516942]: Starting upload Dec 30 05:10:58.676199973 (589h41m58.547283018s) [upload=516942]: Done starting upload Dec 30 05:10:58.676333828 (589h41m58.547416873s) [upload=516942]: Uploading segments... Dec 30 05:10:58.676508070 (589h41m58.547591115s) [upload=516942]: Got next segment. Inline: false Dec 30 05:10:58.676575919 (589h41m58.547658964s) [upload=516942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:10:58.676588103 (589h41m58.547671148s) [upload=516942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:10:58.917398906 (589h41m58.788481941s) [seg_pos={0 0}, upload=516942]: Begin upload segment... Dec 30 05:10:58.917455272 (589h41m58.788538307s) [seg_pos={0 0}, upload=516942]: Done begin upload segment. Dec 30 05:10:58.917481302 (589h41m58.788564337s) [noise=true , address=77.90.31.12:4590 , piece_id=6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.917569400 (589h41m58.788652435s) [noise=true , address=145.79.227.45:4230 , piece_id=RFOYZLJ7BSB6PP3CMQ2SQHKKAUTFSW2AI6737RMDKE2HHACZZVXA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918247341 (589h41m58.789330376s) [noise=true , address=145.79.227.17:4971 , piece_id=DNAVBWG73I6UF76U36YQTMMUJ7J2UP45QGX2SDFVANVNUME7RJZA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918687499 (589h41m58.789770534s) [noise=true , address=45.91.116.14:4191 , piece_id=QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.918975738 (589h41m58.790058783s) [noise=true , address=145.79.227.11:4732 , piece_id=R3QU3O6NDVR5YBEPHI7UMJ63IKAPUJWJY2S2ZE47N5CQCRXFMFFA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.919416117 (589h41m58.790499152s) [noise=true , address=145.79.227.20:4438 , piece_id=I4NRHBOM67LOKR6WO3LNANBBHH6UJVJFH3LNQKJTMA74TCWDYA4Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.919725155 (589h41m58.79080819s) [noise=true , address=45.91.116.11:4117 , piece_id=W7KKEY6AXB7PMVYMDOZNPACFJP5KDSTZGVLT4RQU7H54H2YGF53Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920082305 (589h41m58.79116535s) [noise=true , address=145.79.227.14:4838 , piece_id=NXIMNSYGILEY4V7IXD4RWZW4X7KBF3VKG7OA6ULOEKCVP5VUNNWA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920476976 (589h41m58.791560011s) [noise=true , address=145.79.227.10:4401 , piece_id=52DK6S6UHVTTBNJQDZNJCQCZG4VJCAGU53FRNEZSHEXVGHDEP3WA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.920931001 (589h41m58.792014036s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.922889700 (589h41m58.793972745s) [noise=true , address=145.79.227.6:4258 , piece_id=BQVTINULZFBEO77AFDDZZZINEWVCIKZ4JY5B7LHKBM7EM2VI2I5Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.924179015 (589h41m58.79526205s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.925304348 (589h41m58.796387393s) [noise=true , address=145.79.227.9:4366 , piece_id=SGV3HP3J2RHHR2T6RSKPQH5Z3GGKFSENXVJWYZNQJ5XPRQVZPM4Q, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.926549098 (589h41m58.797632133s) [noise=true , address=145.79.227.4:4186 , piece_id=Y5SEXLAN6GV3SYD4S5MDLKTPTDTTUC6BIFJIK53DL4J2OZYONSGQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.927665103 (589h41m58.798748148s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:10:58.928804724 (589h41m58.799887759s) [noise=true , address=45.91.116.13:4155 , piece_id=L6NQ4ATJU3ORHBEQBMZODIAJG6L25T4RKNOQ5CUMTXLOGQYM67BA, seg_pos={0 0}, upload=516942]: Uploading piece... Dec 30 05:11:00.522237737 (589h42m0.393320782s) [upload=516942]: Next returned nil segment Dec 30 05:11:00.522253286 (589h42m0.393336331s) [upload=516942]: Waiting for error group managing segments... Dec 30 05:11:00.904517749 (589h42m0.775600785s) [noise=true , address=145.79.227.11:4732 , piece_id=R3QU3O6NDVR5YBEPHI7UMJ63IKAPUJWJY2S2ZE47N5CQCRXFMFFA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.908073510 (589h42m0.779156545s) [noise=true , address=145.79.227.14:4838 , piece_id=NXIMNSYGILEY4V7IXD4RWZW4X7KBF3VKG7OA6ULOEKCVP5VUNNWA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.909693365 (589h42m0.7807764s) [noise=true , address=145.79.227.45:4230 , piece_id=RFOYZLJ7BSB6PP3CMQ2SQHKKAUTFSW2AI6737RMDKE2HHACZZVXA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.910894281 (589h42m0.781977316s) [noise=true , address=145.79.227.6:4258 , piece_id=BQVTINULZFBEO77AFDDZZZINEWVCIKZ4JY5B7LHKBM7EM2VI2I5Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.928505957 (589h42m0.799589002s) [noise=true , address=145.79.227.10:4401 , piece_id=52DK6S6UHVTTBNJQDZNJCQCZG4VJCAGU53FRNEZSHEXVGHDEP3WA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.928990931 (589h42m0.800073966s) [noise=true , address=145.79.227.20:4438 , piece_id=I4NRHBOM67LOKR6WO3LNANBBHH6UJVJFH3LNQKJTMA74TCWDYA4Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.929344133 (589h42m0.800427178s) [noise=true , address=145.79.227.17:4971 , piece_id=DNAVBWG73I6UF76U36YQTMMUJ7J2UP45QGX2SDFVANVNUME7RJZA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.936862780 (589h42m0.807945835s) [noise=true , address=145.79.227.9:4366 , piece_id=SGV3HP3J2RHHR2T6RSKPQH5Z3GGKFSENXVJWYZNQJ5XPRQVZPM4Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:00.937960160 (589h42m0.809043205s) [noise=true , address=145.79.227.4:4186 , piece_id=Y5SEXLAN6GV3SYD4S5MDLKTPTDTTUC6BIFJIK53DL4J2OZYONSGQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.090721532 (589h42m0.961804577s) [noise=true , address=77.90.31.12:4590 , piece_id=6EN74NBCKCDKBMH43AQXSTSNQ3SHVM4MYK6ZSSJCYH2UXG5CF3GQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.152347930 (589h42m1.023430975s) [noise=true , address=45.91.116.14:4191 , piece_id=QVLSOO4BNN4O7GSYIQNG3KH7R3K2ELZHSYMPOTLP2YLPBJA63W6Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.304445216 (589h42m1.175528261s) [noise=true , address=45.91.116.11:4117 , piece_id=W7KKEY6AXB7PMVYMDOZNPACFJP5KDSTZGVLT4RQU7H54H2YGF53Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.353503822 (589h42m1.224586867s) [noise=true , address=45.91.116.13:4155 , piece_id=L6NQ4ATJU3ORHBEQBMZODIAJG6L25T4RKNOQ5CUMTXLOGQYM67BA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: <nil> Dec 30 05:11:01.353533188 (589h42m1.224616233s) [seg_pos={0 0}, upload=516942]: Segment reached optimal threshold of 13 pieces. Dec 30 05:11:01.353777343 (589h42m1.224860388s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.353785237 (589h42m1.224868282s) [noise=true , address=45.91.116.6:4975 , piece_id=GFWL3L64AQKRG5FNLZ6TC74HAFHXHQOD4NTRGAOF3WESUYXKZR2Q, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354094436 (589h42m1.225177471s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.354103253 (589h42m1.225186288s) [noise=true , address=145.79.227.49:4339 , piece_id=QT5HCNYCV6H6BAUQJGCJM3BZV7AWQQBITCJDNPFWBG54PLJMR3OQ, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354415377 (589h42m1.225498422s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 05:11:01.354442529 (589h42m1.225525614s) [noise=true , address=45.91.116.10:4083 , piece_id=BBJAICLZMTAKPYAR5V26FM6FYTM7CUS7LJG332OI7FNEX56XUPZA, seg_pos={0 0}, upload=516942]: Long tail cancellation due to optimal threshold Dec 30 05:11:01.354514726 (589h42m1.225597772s) [seg_pos={0 0}, upload=516942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:11:01.354671896 (589h42m1.225754941s) [upload=516942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:11:01.354679841 (589h42m1.225762886s) [upload=516942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:11:01.646260317 (589h42m1.517343382s) [upload=516942]: Done uploading segments... Dec 30 05:11:01.646402247 (589h42m1.517485292s) [upload=516942]: Upload finished. err: <nil> Duration: 30.004795566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.....X.....X.......................] duration: 52.18646106s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:10:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 52.186492189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X.............X...........X............X.X....X...] duration: 5.001000672s trace: https://jaeger.dev.storj.io/trace/6b078ceaf59e0eb7 FAIL (2025-12-30 05:10:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001030479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.XX] duration: 30.005899151s trace: https://jaeger.dev.storj.io/trace/573e8520f01bc4df FAIL (2025-12-30 05:10:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:10:00.973605597 (589h41m0.844688642s) [upload=516925]: Starting upload Dec 30 05:10:00.973660812 (589h41m0.844743867s) [upload=516925]: Done starting upload Dec 30 05:10:00.973775120 (589h41m0.844858165s) [upload=516925]: Uploading segments... Dec 30 05:10:00.973908995 (589h41m0.84499204s) [upload=516925]: Got next segment. Inline: false Dec 30 05:10:00.973960903 (589h41m0.845043948s) [upload=516925]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:10:00.973967106 (589h41m0.845050151s) [upload=516925]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:10:01.223991562 (589h41m1.095074607s) [seg_pos={0 0}, upload=516925]: Begin upload segment... Dec 30 05:10:01.224052849 (589h41m1.095135894s) [seg_pos={0 0}, upload=516925]: Done begin upload segment. Dec 30 05:10:01.224128192 (589h41m1.095211257s) [noise=true , address=145.79.227.47:4267 , piece_id=7SXQTJZ3G7T3MXPA6YHDFRRLA7MPGRB6Y3IWXFPIKULQ6WVEEWHA, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.224213766 (589h41m1.095296811s) [noise=true , address=145.79.227.42:4122 , piece_id=MINYXBW4JHEUXJ7V2OX2YUDSNRP62QABGZW5YGEZGZGBSIPLH6AQ, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.226272957 (589h41m1.097356032s) [noise=true , address=45.91.116.12:4225 , piece_id=2OJSRZYXTB66IIPF7ITXUUVIDXIFJ5CXXD627N4G2YMP3OBQNFHA, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.226882326 (589h41m1.097965371s) [noise=true , address=145.79.227.46:4601 , piece_id=XWBOO35RU74VHOF2BZSWTPIIVQ5H6ZE6TKZ7PHDIAC32M4HZQFHA, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.227319870 (589h41m1.098402915s) [noise=true , address=145.79.227.19:4078 , piece_id=ZRT7TI333VMWTPXNEXPL7CYS5DEYCQPZ4E7URB7LQFOM7S3NMQBQ, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.227388170 (589h41m1.098471215s) [noise=true , address=145.79.227.20:4439 , piece_id=PTDSBKABIVYEBCZ4XYN2BKGTJ7666AXNLUI7TI5EM3OYFCYKK46Q, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.227754497 (589h41m1.098837542s) [noise=true , address=145.79.227.9:4365 , piece_id=ITNWX6LZMZTLRKD72ZLDEKDQ3NOBOW75IJ6IKHBVMGXG2EXVLXYA, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.228194415 (589h41m1.09927746s) [noise=true , address=45.91.116.4:4939 , piece_id=ZA2VERAE3UFFVAPTDLXTZCVNWH7QYBRDFDPPAZTZTQZEOMP2ZQVQ, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.228674229 (589h41m1.099757274s) [noise=true , address=145.79.227.15:4874 , piece_id=NK22XCJZ3Q7OA542Y7HT3JEU2HH2HBFQNBIQ53ZAZUTZAM7CPVFA, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.228730355 (589h41m1.0998134s) [noise=true , address=145.79.227.26:4653 , piece_id=XZB547KSY5EUTUXIQ3CT4BXAPODGY3NR43ZKLYTYQC5ZTNATWHSQ, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.229195111 (589h41m1.100278156s) [noise=true , address=145.79.227.36:4517 , piece_id=NXHZ5IL4MASDWOVHPGOMLUQA5DEMHBGNHHG566ZR4VYVUSOIGLIA, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.229367399 (589h41m1.100450454s) [noise=true , address=45.91.116.9:4264 , piece_id=RDBAG77PXBIPH5R2JPU6LO3QPM6RC5VK26PVRYKWQX4WCUN66A6A, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.230046271 (589h41m1.101129306s) [noise=true , address=145.79.227.37:4565 , piece_id=B63F5LGIZIAZRKXOIN4LGDVNPQQXQFOP2SYCCBFKIN3GNPCXFK3A, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.236052348 (589h41m1.107135393s) [noise=true , address=145.79.227.49:4339 , piece_id=RQ7FBHN6GEOJPSJRH2J7ZB7SCIB4REFCKBLJLWSPWGIOBC2GF3DA, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.237408330 (589h41m1.108491375s) [noise=true , address=77.90.31.17:4770 , piece_id=B4N6G4EH7EMBBWXJIMV7535D6STGWCRLETXPJZUMQUAF5FCPYKJA, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:01.238793758 (589h41m1.109876793s) [noise=true , address=45.91.116.6:4976 , piece_id=O4CJHI5ZI73GGFA2R55DZFH5UA3IO5SJMUSI42REDANE74D37R2A, seg_pos={0 0}, upload=516925]: Uploading piece... Dec 30 05:10:02.869004181 (589h41m2.740087236s) [upload=516925]: Next returned nil segment Dec 30 05:10:02.869065278 (589h41m2.740148343s) [upload=516925]: Waiting for error group managing segments... Dec 30 05:10:03.251999867 (589h41m3.123082922s) [noise=true , address=145.79.227.36:4517 , piece_id=NXHZ5IL4MASDWOVHPGOMLUQA5DEMHBGNHHG566ZR4VYVUSOIGLIA, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.261404414 (589h41m3.13248746s) [noise=true , address=145.79.227.19:4078 , piece_id=ZRT7TI333VMWTPXNEXPL7CYS5DEYCQPZ4E7URB7LQFOM7S3NMQBQ, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.261596811 (589h41m3.132679866s) [noise=true , address=145.79.227.42:4122 , piece_id=MINYXBW4JHEUXJ7V2OX2YUDSNRP62QABGZW5YGEZGZGBSIPLH6AQ, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.264401111 (589h41m3.135484156s) [noise=true , address=145.79.227.37:4565 , piece_id=B63F5LGIZIAZRKXOIN4LGDVNPQQXQFOP2SYCCBFKIN3GNPCXFK3A, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.267183268 (589h41m3.138266313s) [noise=true , address=145.79.227.9:4365 , piece_id=ITNWX6LZMZTLRKD72ZLDEKDQ3NOBOW75IJ6IKHBVMGXG2EXVLXYA, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.284086945 (589h41m3.15516999s) [noise=true , address=145.79.227.15:4874 , piece_id=NK22XCJZ3Q7OA542Y7HT3JEU2HH2HBFQNBIQ53ZAZUTZAM7CPVFA, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.284818759 (589h41m3.155901804s) [noise=true , address=145.79.227.20:4439 , piece_id=PTDSBKABIVYEBCZ4XYN2BKGTJ7666AXNLUI7TI5EM3OYFCYKK46Q, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.289057220 (589h41m3.160140275s) [noise=true , address=145.79.227.46:4601 , piece_id=XWBOO35RU74VHOF2BZSWTPIIVQ5H6ZE6TKZ7PHDIAC32M4HZQFHA, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.291594301 (589h41m3.162677366s) [noise=true , address=145.79.227.47:4267 , piece_id=7SXQTJZ3G7T3MXPA6YHDFRRLA7MPGRB6Y3IWXFPIKULQ6WVEEWHA, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.293557940 (589h41m3.164640985s) [noise=true , address=145.79.227.26:4653 , piece_id=XZB547KSY5EUTUXIQ3CT4BXAPODGY3NR43ZKLYTYQC5ZTNATWHSQ, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.582844768 (589h41m3.453927824s) [noise=true , address=45.91.116.4:4939 , piece_id=ZA2VERAE3UFFVAPTDLXTZCVNWH7QYBRDFDPPAZTZTQZEOMP2ZQVQ, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.660115535 (589h41m3.53119859s) [noise=true , address=45.91.116.12:4225 , piece_id=2OJSRZYXTB66IIPF7ITXUUVIDXIFJ5CXXD627N4G2YMP3OBQNFHA, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.905056562 (589h41m3.776139597s) [noise=true , address=45.91.116.6:4976 , piece_id=O4CJHI5ZI73GGFA2R55DZFH5UA3IO5SJMUSI42REDANE74D37R2A, seg_pos={0 0}, upload=516925]: Done uploading piece. err: <nil> Dec 30 05:10:03.905083793 (589h41m3.776166838s) [seg_pos={0 0}, upload=516925]: Segment reached optimal threshold of 13 pieces. Dec 30 05:10:03.905307400 (589h41m3.776390435s) [noise=true , address=77.90.31.17:4770 , piece_id=B4N6G4EH7EMBBWXJIMV7535D6STGWCRLETXPJZUMQUAF5FCPYKJA, seg_pos={0 0}, upload=516925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:43220->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:43220->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43220->77.90.31.17:4770: use of closed network connection Dec 30 05:10:03.905314433 (589h41m3.776397478s) [noise=true , address=77.90.31.17:4770 , piece_id=B4N6G4EH7EMBBWXJIMV7535D6STGWCRLETXPJZUMQUAF5FCPYKJA, seg_pos={0 0}, upload=516925]: Long tail cancellation due to optimal threshold Dec 30 05:10:03.905625064 (589h41m3.77670811s) [noise=true , address=145.79.227.49:4339 , piece_id=RQ7FBHN6GEOJPSJRH2J7ZB7SCIB4REFCKBLJLWSPWGIOBC2GF3DA, seg_pos={0 0}, upload=516925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 05:10:03.905635955 (589h41m3.776719s) [noise=true , address=145.79.227.49:4339 , piece_id=RQ7FBHN6GEOJPSJRH2J7ZB7SCIB4REFCKBLJLWSPWGIOBC2GF3DA, seg_pos={0 0}, upload=516925]: Long tail cancellation due to optimal threshold Dec 30 05:10:03.905748669 (589h41m3.776831714s) [noise=true , address=45.91.116.9:4264 , piece_id=RDBAG77PXBIPH5R2JPU6LO3QPM6RC5VK26PVRYKWQX4WCUN66A6A, seg_pos={0 0}, upload=516925]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 05:10:03.905754601 (589h41m3.776837646s) [noise=true , address=45.91.116.9:4264 , piece_id=RDBAG77PXBIPH5R2JPU6LO3QPM6RC5VK26PVRYKWQX4WCUN66A6A, seg_pos={0 0}, upload=516925]: Long tail cancellation due to optimal threshold Dec 30 05:10:03.905773567 (589h41m3.776856612s) [seg_pos={0 0}, upload=516925]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:10:03.905909967 (589h41m3.776993012s) [upload=516925]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:10:03.905917080 (589h41m3.777000125s) [upload=516925]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:10:04.206069188 (589h41m4.077152243s) [upload=516925]: Done uploading segments... Dec 30 05:10:04.206213974 (589h41m4.077297049s) [upload=516925]: Upload finished. err: <nil> Duration: 30.005913069s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........X.............X...........X............X.X....X..] duration: 3.611394226s trace: https://jaeger.dev.storj.io/trace/6b078ceaf59e0eb7 FAIL (2025-12-30 05:10:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.611428502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.X] duration: 30.003855535s trace: https://jaeger.dev.storj.io/trace/721c43c28c375613 FAIL (2025-12-30 05:10:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: L6RQAURAT446UCOHRI2MLAISNIVTALAO5OEW24A5KDDWQUATPNLA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: NZNOYXAOJYSJBF7JCCVED6VVNIRXXMJ2YL2TRLWBCSCL6LB4J2BQ) context deadline exceeded ======LOGS====== Dec 30 05:09:30.722206859 (589h40m30.593289894s) [upload=516919]: Starting upload Dec 30 05:09:30.722337758 (589h40m30.593420803s) [upload=516919]: Done starting upload Dec 30 05:09:30.722527419 (589h40m30.593610464s) [upload=516919]: Uploading segments... Dec 30 05:09:30.722581141 (589h40m30.593664176s) [upload=516919]: Got next segment. Inline: false Dec 30 05:09:30.722619344 (589h40m30.593702389s) [upload=516919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:09:30.722626638 (589h40m30.593709674s) [upload=516919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:09:30.969199913 (589h40m30.840282958s) [seg_pos={0 0}, upload=516919]: Begin upload segment... Dec 30 05:09:30.969247934 (589h40m30.840330979s) [seg_pos={0 0}, upload=516919]: Done begin upload segment. Dec 30 05:09:30.969276850 (589h40m30.840359885s) [noise=true , address=145.79.227.17:4970 , piece_id=WPIJ2UVHUQP6FIVZWLTJHKR4NFTXWIRZSJCKHT25GIHFILQZWWXQ, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.969357373 (589h40m30.840440408s) [noise=true , address=45.91.116.5:4298 , piece_id=CMBDMT7LFDAR7RGVC6J2LRRQCLDRJZU5RPIDGZPMLQE2JZQ27GTA, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.969970841 (589h40m30.841053886s) [noise=true , address=145.79.227.14:4838 , piece_id=UG7FLN52AIQGC3CLXI2UK4QIHGNX36LTGLZANN56JWHIBALHJ3LQ, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.970462377 (589h40m30.841545422s) [noise=true , address=145.79.227.49:4339 , piece_id=ZJOSX2XIK6UUXITF6ZKAVJ5MWYJ4GLNWDBLNH4U63S6XCJNMR34A, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.970880233 (589h40m30.841963268s) [noise=true , address=145.79.227.21:4473 , piece_id=RECKJSFI7KDSWOID5DNYPWXOKXPGPVNY7NUZVXVIR54TMPOH4P5A, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.971343636 (589h40m30.842426681s) [noise=true , address=45.91.116.12:4228 , piece_id=NZNOYXAOJYSJBF7JCCVED6VVNIRXXMJ2YL2TRLWBCSCL6LB4J2BQ, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.971789774 (589h40m30.84287282s) [noise=true , address=145.79.227.11:4729 , piece_id=5CSWOPLRIO73GWCRJLRLBQYGXZPOSMIIV7O7W43ORJT2VBMMH4PA, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.972237667 (589h40m30.843320702s) [noise=true , address=145.79.227.38:4978 , piece_id=YAQYBSEOZW7D4UM5MJ3K3RDNTIRXRWMO2XL7UT4DMORRRX4X4JSA, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.973212133 (589h40m30.844295168s) [noise=true , address=145.79.227.7:4293 , piece_id=FCOE5FER3VXZYXMXESD66O6NGXNP7NZO3TES3NH5V4VTCM2XFR3Q, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.973877570 (589h40m30.844960605s) [noise=true , address=145.79.227.3:4151 , piece_id=GYQVYPRPPE353RCRISZ2BLDZHWZMYJ3DIK3BWKJJZ475EIMZCBNA, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.976944109 (589h40m30.848027174s) [noise=true , address=145.79.227.51:4409 , piece_id=TTIRUT5NN6YRXTWO5LKWNRCK6U4ZR4M3RL662AQQH2FI2JNYMGPQ, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.981157783 (589h40m30.852240858s) [noise=true , address=45.91.116.14:4190 , piece_id=L6RQAURAT446UCOHRI2MLAISNIVTALAO5OEW24A5KDDWQUATPNLA, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.984487644 (589h40m30.855570679s) [noise=true , address=45.91.116.6:4973 , piece_id=E4BGPBWTEN5MR477UHJAERJBVIJ266YSENCBKN44FPZVK2MJSDRQ, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.985385674 (589h40m30.856468719s) [noise=true , address=145.79.227.2:4115 , piece_id=IRUEXBXDP4VIG6XZDB6MBL33B5D2GTM7GELS5RZBS3XDHE46OCTA, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.988359807 (589h40m30.859442842s) [noise=true , address=145.79.227.13:4802 , piece_id=IDYUO6JBEIYVGXRDZORLVCWONDIJX4ZXKIUO2LVICAHWMLC6H4HQ, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:30.989954934 (589h40m30.861037979s) [noise=true , address=77.90.31.13:4626 , piece_id=GMDYJ2PCTH5E27H6OVOLDLXI3RB3GS6BBEW52N2JCT3N33F3LBIQ, seg_pos={0 0}, upload=516919]: Uploading piece... Dec 30 05:09:32.681658401 (589h40m32.552741436s) [upload=516919]: Next returned nil segment Dec 30 05:09:32.681672046 (589h40m32.552755081s) [upload=516919]: Waiting for error group managing segments... Dec 30 05:09:32.955901554 (589h40m32.826984589s) [noise=true , address=145.79.227.11:4729 , piece_id=5CSWOPLRIO73GWCRJLRLBQYGXZPOSMIIV7O7W43ORJT2VBMMH4PA, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:32.967554173 (589h40m32.838637228s) [noise=true , address=145.79.227.3:4151 , piece_id=GYQVYPRPPE353RCRISZ2BLDZHWZMYJ3DIK3BWKJJZ475EIMZCBNA, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:32.968243044 (589h40m32.839326089s) [noise=true , address=145.79.227.38:4978 , piece_id=YAQYBSEOZW7D4UM5MJ3K3RDNTIRXRWMO2XL7UT4DMORRRX4X4JSA, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:32.968852635 (589h40m32.83993569s) [noise=true , address=145.79.227.51:4409 , piece_id=TTIRUT5NN6YRXTWO5LKWNRCK6U4ZR4M3RL662AQQH2FI2JNYMGPQ, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:32.970536791 (589h40m32.841619836s) [noise=true , address=145.79.227.7:4293 , piece_id=FCOE5FER3VXZYXMXESD66O6NGXNP7NZO3TES3NH5V4VTCM2XFR3Q, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:32.977550907 (589h40m32.848633942s) [noise=true , address=145.79.227.17:4970 , piece_id=WPIJ2UVHUQP6FIVZWLTJHKR4NFTXWIRZSJCKHT25GIHFILQZWWXQ, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:32.983156481 (589h40m32.854239516s) [noise=true , address=145.79.227.14:4838 , piece_id=UG7FLN52AIQGC3CLXI2UK4QIHGNX36LTGLZANN56JWHIBALHJ3LQ, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:32.992785477 (589h40m32.863868512s) [noise=true , address=145.79.227.13:4802 , piece_id=IDYUO6JBEIYVGXRDZORLVCWONDIJX4ZXKIUO2LVICAHWMLC6H4HQ, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:32.996788068 (589h40m32.867871103s) [noise=true , address=145.79.227.21:4473 , piece_id=RECKJSFI7KDSWOID5DNYPWXOKXPGPVNY7NUZVXVIR54TMPOH4P5A, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:33.156855612 (589h40m33.027938647s) [noise=true , address=77.90.31.13:4626 , piece_id=GMDYJ2PCTH5E27H6OVOLDLXI3RB3GS6BBEW52N2JCT3N33F3LBIQ, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:33.243152515 (589h40m33.11423556s) [noise=true , address=45.91.116.14:4190 , piece_id=L6RQAURAT446UCOHRI2MLAISNIVTALAO5OEW24A5KDDWQUATPNLA, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:33.261115901 (589h40m33.132198936s) [noise=true , address=45.91.116.12:4228 , piece_id=NZNOYXAOJYSJBF7JCCVED6VVNIRXXMJ2YL2TRLWBCSCL6LB4J2BQ, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:33.305406377 (589h40m33.176489412s) [noise=true , address=45.91.116.5:4298 , piece_id=CMBDMT7LFDAR7RGVC6J2LRRQCLDRJZU5RPIDGZPMLQE2JZQ27GTA, seg_pos={0 0}, upload=516919]: Done uploading piece. err: <nil> Dec 30 05:09:33.305425483 (589h40m33.176508518s) [seg_pos={0 0}, upload=516919]: Segment reached optimal threshold of 13 pieces. Dec 30 05:09:33.305618050 (589h40m33.176701085s) [noise=true , address=145.79.227.2:4115 , piece_id=IRUEXBXDP4VIG6XZDB6MBL33B5D2GTM7GELS5RZBS3XDHE46OCTA, seg_pos={0 0}, upload=516919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: expected piece hash; context canceled; EOF Dec 30 05:09:33.305624612 (589h40m33.176707647s) [noise=true , address=145.79.227.2:4115 , piece_id=IRUEXBXDP4VIG6XZDB6MBL33B5D2GTM7GELS5RZBS3XDHE46OCTA, seg_pos={0 0}, upload=516919]: Long tail cancellation due to optimal threshold Dec 30 05:09:33.305636245 (589h40m33.17671928s) [noise=true , address=145.79.227.49:4339 , piece_id=ZJOSX2XIK6UUXITF6ZKAVJ5MWYJ4GLNWDBLNH4U63S6XCJNMR34A, seg_pos={0 0}, upload=516919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 30 05:09:33.305644481 (589h40m33.176727516s) [noise=true , address=145.79.227.49:4339 , piece_id=ZJOSX2XIK6UUXITF6ZKAVJ5MWYJ4GLNWDBLNH4U63S6XCJNMR34A, seg_pos={0 0}, upload=516919]: Long tail cancellation due to optimal threshold Dec 30 05:09:33.305833570 (589h40m33.176916605s) [noise=true , address=45.91.116.6:4973 , piece_id=E4BGPBWTEN5MR477UHJAERJBVIJ266YSENCBKN44FPZVK2MJSDRQ, seg_pos={0 0}, upload=516919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:60302->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 05:09:33.305839692 (589h40m33.176922727s) [noise=true , address=45.91.116.6:4973 , piece_id=E4BGPBWTEN5MR477UHJAERJBVIJ266YSENCBKN44FPZVK2MJSDRQ, seg_pos={0 0}, upload=516919]: Long tail cancellation due to optimal threshold Dec 30 05:09:33.305854330 (589h40m33.176937375s) [seg_pos={0 0}, upload=516919]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:09:33.305918823 (589h40m33.177001858s) [upload=516919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:09:33.305924654 (589h40m33.177007689s) [upload=516919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:09:33.614725041 (589h40m33.485808076s) [upload=516919]: Done uploading segments... Dec 30 05:09:33.614743976 (589h40m33.485827021s) [upload=516919]: Upload finished. err: <nil> Duration: 30.00387842s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.....X.....X......................] duration: 37.993015637s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:09:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 37.993040866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X.............X...........X............X.X....X.] duration: 3.37298612s trace: https://jaeger.dev.storj.io/trace/6b078ceaf59e0eb7 FAIL (2025-12-30 05:09:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.373010245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X.............X...........X............X.X....X] duration: 5.001165664s trace: https://jaeger.dev.storj.io/trace/6b078ceaf59e0eb7 FAIL (2025-12-30 05:09:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001193556s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X.....X.....X.....................] duration: 15.827099875s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:08:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 15.827127738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X..X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX.] duration: 5.984523586s trace: https://jaeger.dev.storj.io/trace/4a64e59d2382e90 FAIL (2025-12-30 05:08:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: NKBLUB3YJ4EOYCXS5K3AIJ2EZG3IZMT7LJX7IYAH2JGR6YBTJVEQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: KFDLB2QMU3R62EJ53EATTVPVCUR2NYKOXX4FNR4BBICBNPT4ZP5A) context deadline exceeded ======LOGS====== c 30 05:07:28.313915982 (589h38m28.184999027s) [upload=516888]: Starting upload Dec 30 05:07:28.313991687 (589h38m28.185074732s) [upload=516888]: Done starting upload Dec 30 05:07:28.314135080 (589h38m28.185218125s) [upload=516888]: Uploading segments... Dec 30 05:07:28.314216835 (589h38m28.185299881s) [upload=516888]: Got next segment. Inline: false Dec 30 05:07:28.314274435 (589h38m28.185357481s) [upload=516888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:07:28.314296006 (589h38m28.185379051s) [upload=516888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:07:28.561462322 (589h38m28.432545367s) [seg_pos={0 0}, upload=516888]: Begin upload segment... Dec 30 05:07:28.561516304 (589h38m28.432599349s) [seg_pos={0 0}, upload=516888]: Done begin upload segment. Dec 30 05:07:28.561561521 (589h38m28.432644566s) [noise=true , address=145.79.227.52:4446 , piece_id=TA4TTY5ZWO4PUQ7LJ4ZSKYVK5EWPSG65XE5BY7DHO6LRGPM6MZBQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.561634660 (589h38m28.432717695s) [noise=true , address=45.91.116.8:4046 , piece_id=QLFSDB4ZJWRMHMVRQO74HOE7OTSVP2RICMUEFYOVGYGW6IPQXZJQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.562243319 (589h38m28.433326364s) [noise=true , address=45.91.116.7:4011 , piece_id=NKBLUB3YJ4EOYCXS5K3AIJ2EZG3IZMT7LJX7IYAH2JGR6YBTJVEQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.562671744 (589h38m28.433754779s) [noise=true , address=145.79.227.50:4374 , piece_id=XQFTCPTC7W4LB2T2BM5ZHBXY6E6RJYLQK3MEQIUNPIEDKVWGVMHQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.562981374 (589h38m28.434064409s) [noise=true , address=77.90.31.10:4517 , piece_id=7LYC6EZSTOKJE7CTHVU56HA66IALDTO4WNESFQT4D6XCENNOWTJA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.563291444 (589h38m28.434374479s) [noise=true , address=145.79.227.22:4509 , piece_id=V6IHIJSCRCUF4OBTZGAMQD4OSBSWZSKBMHPA5JDLWYT7TNOEQY7A, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.563639878 (589h38m28.434722913s) [noise=true , address=77.90.31.21:4878 , piece_id=KFDLB2QMU3R62EJ53EATTVPVCUR2NYKOXX4FNR4BBICBNPT4ZP5A, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.563976489 (589h38m28.435059524s) [noise=true , address=145.79.227.42:4121 , piece_id=ROTQONS3YNPTFQTPHKO4DCOOMFZK2CBGTQNCV47B2KY442S45RTA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.564330854 (589h38m28.435413899s) [noise=true , address=145.79.227.44:4194 , piece_id=PVJPU7PVT65Z5CO6CYBKWYA3KYJRDDY4OXKFS7ESCDS3HY6KCOTQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.566257422 (589h38m28.437340457s) [noise=true , address=145.79.227.28:4725 , piece_id=CMDATZUR5V2FRE4OZQHJMEZEB7R4XHGQF2DLHMWAMUILMUK2X2SA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.566723480 (589h38m28.437806525s) [noise=true , address=45.91.116.14:4189 , piece_id=GCSSR4F42P7754QEODYZCS6PKSLGXENOC3VIMWIJFYT6GY3BFUFQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.568133444 (589h38m28.439216489s) [noise=true , address=145.79.227.14:4839 , piece_id=T5WFSAHWPUVPIGPS3OWHL34FMF4NM5ZMKIIGKV4OC7HS7QXFEHVA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.569535465 (589h38m28.4406185s) [noise=true , address=145.79.227.4:4185 , piece_id=3P2M34MN766W5FR7LB3Z3ERPIVWUMFOQMGQRGDMFNEFXVRSNR7PA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.571060989 (589h38m28.442144034s) [noise=true , address=145.79.227.36:4518 , piece_id=R2NSEFS6P2UVXBT3BFIZ4CXQ3LVRXTTKVJQYZDGU6L5RFRJQRUYA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.572372496 (589h38m28.443455531s) [noise=true , address=145.79.227.38:4978 , piece_id=VWDIWIOBO3JOL3UMDFUOFH5UCPZJ4XGGLFAXKLDR7DVHDX3FDMCA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.573967794 (589h38m28.445050839s) [noise=true , address=77.90.31.14:4664 , piece_id=MRMQCSLD5DXXXL3GSVAHCKIQI7MD45Z4PS6SMZJQFMERJKQIZGUA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:30.268467625 (589h38m30.13955067s) [upload=516888]: Next returned nil segment Dec 30 05:07:30.268486361 (589h38m30.139569406s) [upload=516888]: Waiting for error group managing segments... Dec 30 05:07:30.549188052 (589h38m30.420271087s) [noise=true , address=145.79.227.28:4725 , piece_id=CMDATZUR5V2FRE4OZQHJMEZEB7R4XHGQF2DLHMWAMUILMUK2X2SA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.549458317 (589h38m30.420541362s) [noise=true , address=145.79.227.4:4185 , piece_id=3P2M34MN766W5FR7LB3Z3ERPIVWUMFOQMGQRGDMFNEFXVRSNR7PA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.549949993 (589h38m30.421033028s) [noise=true , address=145.79.227.14:4839 , piece_id=T5WFSAHWPUVPIGPS3OWHL34FMF4NM5ZMKIIGKV4OC7HS7QXFEHVA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.554424293 (589h38m30.425507338s) [noise=true , address=145.79.227.50:4374 , piece_id=XQFTCPTC7W4LB2T2BM5ZHBXY6E6RJYLQK3MEQIUNPIEDKVWGVMHQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.554628391 (589h38m30.425711426s) [noise=true , address=145.79.227.22:4509 , piece_id=V6IHIJSCRCUF4OBTZGAMQD4OSBSWZSKBMHPA5JDLWYT7TNOEQY7A, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.563736705 (589h38m30.43481974s) [noise=true , address=145.79.227.36:4518 , piece_id=R2NSEFS6P2UVXBT3BFIZ4CXQ3LVRXTTKVJQYZDGU6L5RFRJQRUYA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.566766975 (589h38m30.43785002s) [noise=true , address=145.79.227.38:4978 , piece_id=VWDIWIOBO3JOL3UMDFUOFH5UCPZJ4XGGLFAXKLDR7DVHDX3FDMCA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.574049843 (589h38m30.445132978s) [noise=true , address=145.79.227.42:4121 , piece_id=ROTQONS3YNPTFQTPHKO4DCOOMFZK2CBGTQNCV47B2KY442S45RTA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.576269410 (589h38m30.447352445s) [noise=true , address=145.79.227.52:4446 , piece_id=TA4TTY5ZWO4PUQ7LJ4ZSKYVK5EWPSG65XE5BY7DHO6LRGPM6MZBQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.739332317 (589h38m30.610415362s) [noise=true , address=77.90.31.21:4878 , piece_id=KFDLB2QMU3R62EJ53EATTVPVCUR2NYKOXX4FNR4BBICBNPT4ZP5A, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.872055524 (589h38m30.743138559s) [noise=true , address=77.90.31.10:4517 , piece_id=7LYC6EZSTOKJE7CTHVU56HA66IALDTO4WNESFQT4D6XCENNOWTJA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.879634205 (589h38m30.75071725s) [noise=true , address=45.91.116.8:4046 , piece_id=QLFSDB4ZJWRMHMVRQO74HOE7OTSVP2RICMUEFYOVGYGW6IPQXZJQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:31.168405031 (589h38m31.039488076s) [noise=true , address=45.91.116.7:4011 , piece_id=NKBLUB3YJ4EOYCXS5K3AIJ2EZG3IZMT7LJX7IYAH2JGR6YBTJVEQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:31.168434998 (589h38m31.039518053s) [seg_pos={0 0}, upload=516888]: Segment reached optimal threshold of 13 pieces. Dec 30 05:07:31.168691867 (589h38m31.039774922s) [noise=true , address=45.91.116.14:4189 , piece_id=GCSSR4F42P7754QEODYZCS6PKSLGXENOC3VIMWIJFYT6GY3BFUFQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 05:07:31.168701376 (589h38m31.039784421s) [noise=true , address=45.91.116.14:4189 , piece_id=GCSSR4F42P7754QEODYZCS6PKSLGXENOC3VIMWIJFYT6GY3BFUFQ, seg_pos={0 0}, upload=516888]: Long tail cancellation due to optimal threshold Dec 30 05:07:31.168980848 (589h38m31.040063893s) [noise=true , address=77.90.31.14:4664 , piece_id=MRMQCSLD5DXXXL3GSVAHCKIQI7MD45Z4PS6SMZJQFMERJKQIZGUA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:48426->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:48426->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48426->77.90.31.14:4664: use of closed network connection Dec 30 05:07:31.168989244 (589h38m31.040072279s) [noise=true , address=77.90.31.14:4664 , piece_id=MRMQCSLD5DXXXL3GSVAHCKIQI7MD45Z4PS6SMZJQFMERJKQIZGUA, seg_pos={0 0}, upload=516888]: Long tail cancellation due to optimal threshold Dec 30 05:07:31.169165589 (589h38m31.040248634s) [noise=true , address=145.79.227.44:4194 , piece_id=PVJPU7PVT65Z5CO6CYBKWYA3KYJRDDY4OXKFS7ESCDS3HY6KCOTQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 05:07:31.169195637 (589h38m31.040278702s) [noise=true , address=145.79.227.44:4194 , piece_id=PVJPU7PVT65Z5CO6CYBKWYA3KYJRDDY4OXKFS7ESCDS3HY6KCOTQ, seg_pos={0 0}, upload=516888]: Long tail cancellation due to optimal threshold Dec 30 05:07:31.169253417 (589h38m31.040336452s) [seg_pos={0 0}, upload=516888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:07:31.169382422 (589h38m31.040465467s) [upload=516888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:07:31.169391009 (589h38m31.040474044s) [upload=516888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:07:31.463442981 (589h38m31.334526026s) [upload=516888]: Done uploading segments... Dec 30 05:07:31.463466646 (589h38m31.334549681s) [upload=516888]: Upload finished. err: <nil> Duration: 5.984556799s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.....X.....X....................] duration: 39.05505188s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:08:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 39.055071448s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..............X.............X...........X............X.X....] duration: 4.918138727s trace: https://jaeger.dev.storj.io/trace/107f5b64bd5ac16b FAIL (2025-12-30 05:08:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.918166741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X..X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..XX] duration: 30.004698464s trace: https://jaeger.dev.storj.io/trace/4a64e59d2382e90 FAIL (2025-12-30 05:07:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: NKBLUB3YJ4EOYCXS5K3AIJ2EZG3IZMT7LJX7IYAH2JGR6YBTJVEQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: KFDLB2QMU3R62EJ53EATTVPVCUR2NYKOXX4FNR4BBICBNPT4ZP5A) context deadline exceeded ======LOGS====== c 30 05:07:28.313915982 (589h38m28.184999027s) [upload=516888]: Starting upload Dec 30 05:07:28.313991687 (589h38m28.185074732s) [upload=516888]: Done starting upload Dec 30 05:07:28.314135080 (589h38m28.185218125s) [upload=516888]: Uploading segments... Dec 30 05:07:28.314216835 (589h38m28.185299881s) [upload=516888]: Got next segment. Inline: false Dec 30 05:07:28.314274435 (589h38m28.185357481s) [upload=516888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:07:28.314296006 (589h38m28.185379051s) [upload=516888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:07:28.561462322 (589h38m28.432545367s) [seg_pos={0 0}, upload=516888]: Begin upload segment... Dec 30 05:07:28.561516304 (589h38m28.432599349s) [seg_pos={0 0}, upload=516888]: Done begin upload segment. Dec 30 05:07:28.561561521 (589h38m28.432644566s) [noise=true , address=145.79.227.52:4446 , piece_id=TA4TTY5ZWO4PUQ7LJ4ZSKYVK5EWPSG65XE5BY7DHO6LRGPM6MZBQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.561634660 (589h38m28.432717695s) [noise=true , address=45.91.116.8:4046 , piece_id=QLFSDB4ZJWRMHMVRQO74HOE7OTSVP2RICMUEFYOVGYGW6IPQXZJQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.562243319 (589h38m28.433326364s) [noise=true , address=45.91.116.7:4011 , piece_id=NKBLUB3YJ4EOYCXS5K3AIJ2EZG3IZMT7LJX7IYAH2JGR6YBTJVEQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.562671744 (589h38m28.433754779s) [noise=true , address=145.79.227.50:4374 , piece_id=XQFTCPTC7W4LB2T2BM5ZHBXY6E6RJYLQK3MEQIUNPIEDKVWGVMHQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.562981374 (589h38m28.434064409s) [noise=true , address=77.90.31.10:4517 , piece_id=7LYC6EZSTOKJE7CTHVU56HA66IALDTO4WNESFQT4D6XCENNOWTJA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.563291444 (589h38m28.434374479s) [noise=true , address=145.79.227.22:4509 , piece_id=V6IHIJSCRCUF4OBTZGAMQD4OSBSWZSKBMHPA5JDLWYT7TNOEQY7A, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.563639878 (589h38m28.434722913s) [noise=true , address=77.90.31.21:4878 , piece_id=KFDLB2QMU3R62EJ53EATTVPVCUR2NYKOXX4FNR4BBICBNPT4ZP5A, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.563976489 (589h38m28.435059524s) [noise=true , address=145.79.227.42:4121 , piece_id=ROTQONS3YNPTFQTPHKO4DCOOMFZK2CBGTQNCV47B2KY442S45RTA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.564330854 (589h38m28.435413899s) [noise=true , address=145.79.227.44:4194 , piece_id=PVJPU7PVT65Z5CO6CYBKWYA3KYJRDDY4OXKFS7ESCDS3HY6KCOTQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.566257422 (589h38m28.437340457s) [noise=true , address=145.79.227.28:4725 , piece_id=CMDATZUR5V2FRE4OZQHJMEZEB7R4XHGQF2DLHMWAMUILMUK2X2SA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.566723480 (589h38m28.437806525s) [noise=true , address=45.91.116.14:4189 , piece_id=GCSSR4F42P7754QEODYZCS6PKSLGXENOC3VIMWIJFYT6GY3BFUFQ, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.568133444 (589h38m28.439216489s) [noise=true , address=145.79.227.14:4839 , piece_id=T5WFSAHWPUVPIGPS3OWHL34FMF4NM5ZMKIIGKV4OC7HS7QXFEHVA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.569535465 (589h38m28.4406185s) [noise=true , address=145.79.227.4:4185 , piece_id=3P2M34MN766W5FR7LB3Z3ERPIVWUMFOQMGQRGDMFNEFXVRSNR7PA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.571060989 (589h38m28.442144034s) [noise=true , address=145.79.227.36:4518 , piece_id=R2NSEFS6P2UVXBT3BFIZ4CXQ3LVRXTTKVJQYZDGU6L5RFRJQRUYA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.572372496 (589h38m28.443455531s) [noise=true , address=145.79.227.38:4978 , piece_id=VWDIWIOBO3JOL3UMDFUOFH5UCPZJ4XGGLFAXKLDR7DVHDX3FDMCA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:28.573967794 (589h38m28.445050839s) [noise=true , address=77.90.31.14:4664 , piece_id=MRMQCSLD5DXXXL3GSVAHCKIQI7MD45Z4PS6SMZJQFMERJKQIZGUA, seg_pos={0 0}, upload=516888]: Uploading piece... Dec 30 05:07:30.268467625 (589h38m30.13955067s) [upload=516888]: Next returned nil segment Dec 30 05:07:30.268486361 (589h38m30.139569406s) [upload=516888]: Waiting for error group managing segments... Dec 30 05:07:30.549188052 (589h38m30.420271087s) [noise=true , address=145.79.227.28:4725 , piece_id=CMDATZUR5V2FRE4OZQHJMEZEB7R4XHGQF2DLHMWAMUILMUK2X2SA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.549458317 (589h38m30.420541362s) [noise=true , address=145.79.227.4:4185 , piece_id=3P2M34MN766W5FR7LB3Z3ERPIVWUMFOQMGQRGDMFNEFXVRSNR7PA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.549949993 (589h38m30.421033028s) [noise=true , address=145.79.227.14:4839 , piece_id=T5WFSAHWPUVPIGPS3OWHL34FMF4NM5ZMKIIGKV4OC7HS7QXFEHVA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.554424293 (589h38m30.425507338s) [noise=true , address=145.79.227.50:4374 , piece_id=XQFTCPTC7W4LB2T2BM5ZHBXY6E6RJYLQK3MEQIUNPIEDKVWGVMHQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.554628391 (589h38m30.425711426s) [noise=true , address=145.79.227.22:4509 , piece_id=V6IHIJSCRCUF4OBTZGAMQD4OSBSWZSKBMHPA5JDLWYT7TNOEQY7A, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.563736705 (589h38m30.43481974s) [noise=true , address=145.79.227.36:4518 , piece_id=R2NSEFS6P2UVXBT3BFIZ4CXQ3LVRXTTKVJQYZDGU6L5RFRJQRUYA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.566766975 (589h38m30.43785002s) [noise=true , address=145.79.227.38:4978 , piece_id=VWDIWIOBO3JOL3UMDFUOFH5UCPZJ4XGGLFAXKLDR7DVHDX3FDMCA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.574049843 (589h38m30.445132978s) [noise=true , address=145.79.227.42:4121 , piece_id=ROTQONS3YNPTFQTPHKO4DCOOMFZK2CBGTQNCV47B2KY442S45RTA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.576269410 (589h38m30.447352445s) [noise=true , address=145.79.227.52:4446 , piece_id=TA4TTY5ZWO4PUQ7LJ4ZSKYVK5EWPSG65XE5BY7DHO6LRGPM6MZBQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.739332317 (589h38m30.610415362s) [noise=true , address=77.90.31.21:4878 , piece_id=KFDLB2QMU3R62EJ53EATTVPVCUR2NYKOXX4FNR4BBICBNPT4ZP5A, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.872055524 (589h38m30.743138559s) [noise=true , address=77.90.31.10:4517 , piece_id=7LYC6EZSTOKJE7CTHVU56HA66IALDTO4WNESFQT4D6XCENNOWTJA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:30.879634205 (589h38m30.75071725s) [noise=true , address=45.91.116.8:4046 , piece_id=QLFSDB4ZJWRMHMVRQO74HOE7OTSVP2RICMUEFYOVGYGW6IPQXZJQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:31.168405031 (589h38m31.039488076s) [noise=true , address=45.91.116.7:4011 , piece_id=NKBLUB3YJ4EOYCXS5K3AIJ2EZG3IZMT7LJX7IYAH2JGR6YBTJVEQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: <nil> Dec 30 05:07:31.168434998 (589h38m31.039518053s) [seg_pos={0 0}, upload=516888]: Segment reached optimal threshold of 13 pieces. Dec 30 05:07:31.168691867 (589h38m31.039774922s) [noise=true , address=45.91.116.14:4189 , piece_id=GCSSR4F42P7754QEODYZCS6PKSLGXENOC3VIMWIJFYT6GY3BFUFQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 05:07:31.168701376 (589h38m31.039784421s) [noise=true , address=45.91.116.14:4189 , piece_id=GCSSR4F42P7754QEODYZCS6PKSLGXENOC3VIMWIJFYT6GY3BFUFQ, seg_pos={0 0}, upload=516888]: Long tail cancellation due to optimal threshold Dec 30 05:07:31.168980848 (589h38m31.040063893s) [noise=true , address=77.90.31.14:4664 , piece_id=MRMQCSLD5DXXXL3GSVAHCKIQI7MD45Z4PS6SMZJQFMERJKQIZGUA, seg_pos={0 0}, upload=516888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:48426->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:48426->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48426->77.90.31.14:4664: use of closed network connection Dec 30 05:07:31.168989244 (589h38m31.040072279s) [noise=true , address=77.90.31.14:4664 , piece_id=MRMQCSLD5DXXXL3GSVAHCKIQI7MD45Z4PS6SMZJQFMERJKQIZGUA, seg_pos={0 0}, upload=516888]: Long tail cancellation due to optimal threshold Dec 30 05:07:31.169165589 (589h38m31.040248634s) [noise=true , address=145.79.227.44:4194 , piece_id=PVJPU7PVT65Z5CO6CYBKWYA3KYJRDDY4OXKFS7ESCDS3HY6KCOTQ, seg_pos={0 0}, upload=516888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 30 05:07:31.169195637 (589h38m31.040278702s) [noise=true , address=145.79.227.44:4194 , piece_id=PVJPU7PVT65Z5CO6CYBKWYA3KYJRDDY4OXKFS7ESCDS3HY6KCOTQ, seg_pos={0 0}, upload=516888]: Long tail cancellation due to optimal threshold Dec 30 05:07:31.169253417 (589h38m31.040336452s) [seg_pos={0 0}, upload=516888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:07:31.169382422 (589h38m31.040465467s) [upload=516888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:07:31.169391009 (589h38m31.040474044s) [upload=516888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:07:31.463442981 (589h38m31.334526026s) [upload=516888]: Done uploading segments... Dec 30 05:07:31.463466646 (589h38m31.334549681s) [upload=516888]: Upload finished. err: <nil> Duration: 30.004721669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.....X.....X...................] duration: 43.708559635s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:07:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 43.708589633s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...............X.............X...........X............X.X...] duration: 3.708180875s trace: https://jaeger.dev.storj.io/trace/107f5b64bd5ac16b FAIL (2025-12-30 05:07:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.708207285s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X.............X...........X............X.X..] duration: 4.209053265s trace: https://jaeger.dev.storj.io/trace/107f5b64bd5ac16b FAIL (2025-12-30 05:07:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.209089395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.....X.....X..................] duration: 40.574135682s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:06:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 40.574164988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.X..X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..X] duration: 30.003215755s trace: https://jaeger.dev.storj.io/trace/724599fc776fb531 FAIL (2025-12-30 05:06:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: UTLQZAENGCO35JDGFBJJDXQIO2B7QI4YPC6E7TFV6S5P7T6H5LTA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: RAMWHGMS7ZBRPSCJWOSGHISVTKFMH3WQK4MY4KCORTNM4HDVYIOQ) context deadline exceeded ======LOGS====== Dec 30 05:06:21.016412136 (589h37m20.887495171s) [upload=516872]: Starting upload Dec 30 05:06:21.016471018 (589h37m20.887554053s) [upload=516872]: Done starting upload Dec 30 05:06:21.016676209 (589h37m20.887759244s) [upload=516872]: Uploading segments... Dec 30 05:06:21.016721425 (589h37m20.88780447s) [upload=516872]: Got next segment. Inline: false Dec 30 05:06:21.016756953 (589h37m20.887839998s) [upload=516872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:06:21.016765619 (589h37m20.887848664s) [upload=516872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:06:21.264329572 (589h37m21.135412627s) [seg_pos={0 0}, upload=516872]: Begin upload segment... Dec 30 05:06:21.264398093 (589h37m21.135481138s) [seg_pos={0 0}, upload=516872]: Done begin upload segment. Dec 30 05:06:21.264437387 (589h37m21.135520422s) [noise=true , address=145.79.227.52:4447 , piece_id=D67JCUBLWVJMNTWQSZP6OJTSUK65VDG5JQWIWM3TFRIJZQLOL7GQ, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.264451004 (589h37m21.135534039s) [noise=true , address=77.90.31.19:4844 , piece_id=ROAD36BRMXCFGZRVNYZ33RV32NTNANHIRJN47ZPIWWFPECRIG5IA, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.265030757 (589h37m21.136113802s) [noise=true , address=145.79.227.17:4972 , piece_id=LSNLJBSVD45HVWNK4UK24MX6IWQQKD35TNMTCJDUFLYZ3XQNNUXQ, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.265968763 (589h37m21.137051798s) [noise=true , address=145.79.227.35:4483 , piece_id=PD2UIYICFQN4V2GN4536OEHRV5AUZV7UYAW22POEZMSPIFY76FHA, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.266196938 (589h37m21.137279973s) [noise=true , address=145.79.227.42:4121 , piece_id=ZR3A2YBUAJSZXP2GQWOLPSRQBK4GKW5LTPEOTDHGM5NASGEANAZQ, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.267069560 (589h37m21.138152605s) [noise=true , address=145.79.227.14:4839 , piece_id=THKCKPD6EDBSNSVHZZHE7QLUIOULPKOLFQA6N3JW6OL65RAMY4KQ, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.268255046 (589h37m21.139338091s) [noise=true , address=77.90.31.17:4770 , piece_id=Z3XK47W6CUI2OTI7Y7MU2WPOJVRKBY5N3XHNML7CAN7B2THG3HRQ, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.269972567 (589h37m21.141055612s) [noise=true , address=145.79.227.40:4051 , piece_id=7543NAMBBXKJO3FPDXYBFBREQUJ3N22JIGWH2E6RJJ7Y7N4HOJ7A, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.270235868 (589h37m21.141318903s) [noise=true , address=77.90.31.21:4878 , piece_id=VX36YBFDFDMWXU6RB3NOX5DNREMQRGLZWWGOYUUJ2PWMQWL6VCAA, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.270871298 (589h37m21.141954333s) [noise=true , address=145.79.227.44:4193 , piece_id=AMWQW6B3BS25BLVN2MF7SXS5IE4OWDLPN7CNG36WTNMEYKNNSTIA, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.271934633 (589h37m21.143017668s) [noise=true , address=45.91.116.4:4940 , piece_id=WZEBPCCBAY2AYGVCBEFIPTPWI6T64A3TT7KFIZQ7KKQLT3K4EAOQ, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.272417653 (589h37m21.143500688s) [noise=true , address=45.91.116.10:4081 , piece_id=UTLQZAENGCO35JDGFBJJDXQIO2B7QI4YPC6E7TFV6S5P7T6H5LTA, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.273063722 (589h37m21.144146757s) [noise=true , address=77.90.31.13:4627 , piece_id=Y4NYGLLMKGXW2BZ4HGJPTUTR5XIGOE3WOBHUV7EXW4D45FC6MW3A, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.275231681 (589h37m21.146314716s) [noise=true , address=145.79.227.8:4329 , piece_id=QTBK4NFLO4GYF7DOXE2DEVYZKKWVXZEOM4XUUCIR36L46HYD2UFQ, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.279572767 (589h37m21.150655812s) [noise=true , address=145.79.227.22:4509 , piece_id=ISBVIVNXA64VFU7ID4DW5CWMZWDJNHPZ5LDF7HLRG4T7ZUNIGD2A, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:21.284676395 (589h37m21.15575943s) [noise=true , address=45.91.116.13:4153 , piece_id=RAMWHGMS7ZBRPSCJWOSGHISVTKFMH3WQK4MY4KCORTNM4HDVYIOQ, seg_pos={0 0}, upload=516872]: Uploading piece... Dec 30 05:06:22.861567399 (589h37m22.732650434s) [upload=516872]: Next returned nil segment Dec 30 05:06:22.861874945 (589h37m22.73295798s) [upload=516872]: Waiting for error group managing segments... Dec 30 05:06:23.249601067 (589h37m23.120684102s) [noise=true , address=145.79.227.17:4972 , piece_id=LSNLJBSVD45HVWNK4UK24MX6IWQQKD35TNMTCJDUFLYZ3XQNNUXQ, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.252481062 (589h37m23.123564097s) [noise=true , address=145.79.227.8:4329 , piece_id=QTBK4NFLO4GYF7DOXE2DEVYZKKWVXZEOM4XUUCIR36L46HYD2UFQ, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.257947691 (589h37m23.129030736s) [noise=true , address=145.79.227.42:4121 , piece_id=ZR3A2YBUAJSZXP2GQWOLPSRQBK4GKW5LTPEOTDHGM5NASGEANAZQ, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.258784304 (589h37m23.129867349s) [noise=true , address=145.79.227.14:4839 , piece_id=THKCKPD6EDBSNSVHZZHE7QLUIOULPKOLFQA6N3JW6OL65RAMY4KQ, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.300306328 (589h37m23.171389373s) [noise=true , address=145.79.227.52:4447 , piece_id=D67JCUBLWVJMNTWQSZP6OJTSUK65VDG5JQWIWM3TFRIJZQLOL7GQ, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.301004198 (589h37m23.172087233s) [noise=true , address=145.79.227.22:4509 , piece_id=ISBVIVNXA64VFU7ID4DW5CWMZWDJNHPZ5LDF7HLRG4T7ZUNIGD2A, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.470799575 (589h37m23.34188261s) [noise=true , address=77.90.31.17:4770 , piece_id=Z3XK47W6CUI2OTI7Y7MU2WPOJVRKBY5N3XHNML7CAN7B2THG3HRQ, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.521626528 (589h37m23.392709563s) [noise=true , address=45.91.116.4:4940 , piece_id=WZEBPCCBAY2AYGVCBEFIPTPWI6T64A3TT7KFIZQ7KKQLT3K4EAOQ, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.542829082 (589h37m23.413912127s) [noise=true , address=45.91.116.13:4153 , piece_id=RAMWHGMS7ZBRPSCJWOSGHISVTKFMH3WQK4MY4KCORTNM4HDVYIOQ, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.572379229 (589h37m23.443462274s) [noise=true , address=77.90.31.21:4878 , piece_id=VX36YBFDFDMWXU6RB3NOX5DNREMQRGLZWWGOYUUJ2PWMQWL6VCAA, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.579241075 (589h37m23.45032412s) [noise=true , address=45.91.116.10:4081 , piece_id=UTLQZAENGCO35JDGFBJJDXQIO2B7QI4YPC6E7TFV6S5P7T6H5LTA, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.607675368 (589h37m23.478758403s) [noise=true , address=77.90.31.13:4627 , piece_id=Y4NYGLLMKGXW2BZ4HGJPTUTR5XIGOE3WOBHUV7EXW4D45FC6MW3A, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.634536387 (589h37m23.505619432s) [noise=true , address=77.90.31.19:4844 , piece_id=ROAD36BRMXCFGZRVNYZ33RV32NTNANHIRJN47ZPIWWFPECRIG5IA, seg_pos={0 0}, upload=516872]: Done uploading piece. err: <nil> Dec 30 05:06:23.634562365 (589h37m23.5056454s) [seg_pos={0 0}, upload=516872]: Segment reached optimal threshold of 13 pieces. Dec 30 05:06:23.634805298 (589h37m23.505888333s) [noise=true , address=145.79.227.35:4483 , piece_id=PD2UIYICFQN4V2GN4536OEHRV5AUZV7UYAW22POEZMSPIFY76FHA, seg_pos={0 0}, upload=516872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 30 05:06:23.634814355 (589h37m23.50589739s) [noise=true , address=145.79.227.35:4483 , piece_id=PD2UIYICFQN4V2GN4536OEHRV5AUZV7UYAW22POEZMSPIFY76FHA, seg_pos={0 0}, upload=516872]: Long tail cancellation due to optimal threshold Dec 30 05:06:23.634944663 (589h37m23.506027708s) [noise=true , address=145.79.227.44:4193 , piece_id=AMWQW6B3BS25BLVN2MF7SXS5IE4OWDLPN7CNG36WTNMEYKNNSTIA, seg_pos={0 0}, upload=516872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 30 05:06:23.634985030 (589h37m23.506068085s) [noise=true , address=145.79.227.44:4193 , piece_id=AMWQW6B3BS25BLVN2MF7SXS5IE4OWDLPN7CNG36WTNMEYKNNSTIA, seg_pos={0 0}, upload=516872]: Long tail cancellation due to optimal threshold Dec 30 05:06:23.635091342 (589h37m23.506174377s) [noise=true , address=145.79.227.40:4051 , piece_id=7543NAMBBXKJO3FPDXYBFBREQUJ3N22JIGWH2E6RJJ7Y7N4HOJ7A, seg_pos={0 0}, upload=516872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 05:06:23.635101052 (589h37m23.506184087s) [noise=true , address=145.79.227.40:4051 , piece_id=7543NAMBBXKJO3FPDXYBFBREQUJ3N22JIGWH2E6RJJ7Y7N4HOJ7A, seg_pos={0 0}, upload=516872]: Long tail cancellation due to optimal threshold Dec 30 05:06:23.635120218 (589h37m23.506203253s) [seg_pos={0 0}, upload=516872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:06:23.635210650 (589h37m23.506293685s) [upload=516872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:06:23.635216621 (589h37m23.506299666s) [upload=516872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:06:23.938980267 (589h37m23.810063312s) [upload=516872]: Done uploading segments... Dec 30 05:06:23.939004813 (589h37m23.810087848s) [upload=516872]: Upload finished. err: <nil> Duration: 30.003241424s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [X................X.............X...........X............X.X.] duration: 3.477208496s trace: https://jaeger.dev.storj.io/trace/107f5b64bd5ac16b FAIL (2025-12-30 05:06:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.477240205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.....X.....X.................] duration: 44.077184761s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:05:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 44.07721612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.X................X.............X...........X............X.X] duration: 5.001155013s trace: https://jaeger.dev.storj.io/trace/107f5b64bd5ac16b FAIL (2025-12-30 05:05:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00117907s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.X..X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X..] duration: 5.639274167s trace: https://jaeger.dev.storj.io/trace/68c837d837387aed FAIL (2025-12-30 05:05:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: YJ2JOLQC3CJVCXFIWJNFFZOBUUPTCHSMWZPJSMLPQDSBNO7DHGIQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 2VEY4ZK63THG76APKXXXS6S2JO5U5WUDTGC2EKEHRMTXPNLAUS6Q) context deadline exceeded ======LOGS====== : Got next segment. Inline: false Dec 30 05:03:58.486745661 (589h34m58.357828707s) [upload=516842]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:03:58.486751162 (589h34m58.357834207s) [upload=516842]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:03:58.735896835 (589h34m58.60697987s) [seg_pos={0 0}, upload=516842]: Begin upload segment... Dec 30 05:03:58.735963101 (589h34m58.607046136s) [seg_pos={0 0}, upload=516842]: Done begin upload segment. Dec 30 05:03:58.735980515 (589h34m58.60706356s) [noise=true , address=45.91.116.14:4189 , piece_id=3IKCTH3X6ZY3ZKWXP6RTBUZBARMKTOTXOD2QVD7XLZWVMAFMRKZA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.735998609 (589h34m58.607081654s) [noise=true , address=145.79.227.48:4301 , piece_id=PAPDGXBEA25JOVLRIXZJRRION4AEYFMXWLZDX6W7DBAYYTRI2ZOA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.736069213 (589h34m58.607152258s) [noise=true , address=145.79.227.38:4978 , piece_id=SZBMOYADQXARU6ZKS6CXJ4ZXLJHXGUFHUW3VRL6ND3FIUFLK2KTQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.736687250 (589h34m58.607770295s) [noise=true , address=145.79.227.18:4006 , piece_id=4R2IA577WNTKXY4TXIQ77SBULCSYTO7GYJYTSIAE4C4BI5FPP7QA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.737170360 (589h34m58.608253405s) [noise=true , address=145.79.227.3:4151 , piece_id=PXEEJSLYQAZZPDZ4YIOOYA2ID7M42ODUCRQUYX35AZ26LIPNFLLQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.737573177 (589h34m58.608656212s) [noise=true , address=145.79.227.15:4876 , piece_id=TZ53FFSVWCKUIBAYPXP7JGG6MLFMR2MKKUGNAKY47NQS27HQE5ZQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.738027201 (589h34m58.609110246s) [noise=true , address=45.91.116.11:4117 , piece_id=AZ7ONXIOJBAAWL3K3PEUDN3EHWHYC3HAOFIT6HA4XUBIBGGV4K5Q, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.738543905 (589h34m58.60962694s) [noise=true , address=145.79.227.28:4726 , piece_id=OM4TIWZOREGZMUAI3EUJ2LRZ6EWA6Y6FQUXNT7DD4OUHLIM67CWA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.739062273 (589h34m58.610145318s) [noise=true , address=45.91.116.13:4155 , piece_id=2VEY4ZK63THG76APKXXXS6S2JO5U5WUDTGC2EKEHRMTXPNLAUS6Q, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.739626457 (589h34m58.610709502s) [noise=true , address=145.79.227.5:4222 , piece_id=ZIMAEXD7XAYAEG2NVDFTBHGDYRKKLX5WMYMKQEKIEXJLWTKFVUFA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.747476504 (589h34m58.618559549s) [noise=true , address=145.79.227.52:4446 , piece_id=ZJ5HILORUROTEYZJJY3GCI4K563M6TB4VGM47KVSYU5SIWIRF6YA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.748958116 (589h34m58.620041151s) [noise=true , address=145.79.227.13:4803 , piece_id=LITYLR5VA3WFJ7UTLPRFAEQDBQALNSTSRQDYFOEMNYGPE3NEJQZA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.751521547 (589h34m58.622604592s) [noise=true , address=45.91.116.8:4045 , piece_id=YJ2JOLQC3CJVCXFIWJNFFZOBUUPTCHSMWZPJSMLPQDSBNO7DHGIQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.754403955 (589h34m58.62548699s) [noise=true , address=45.91.116.7:4009 , piece_id=XP2NS6CMVS5TEHHRRL534ZXYVYTARUZJR44BQPDG3G24YS5F4MBA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.755931414 (589h34m58.627014449s) [noise=true , address=77.90.31.10:4520 , piece_id=KDVDPUDXPJYWKITP6YH5RBGPKIFQHX2VTN5BNDOE3AJPEBTLHG4Q, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.756599896 (589h34m58.627682932s) [noise=true , address=145.79.227.16:4911 , piece_id=2JOIF5B3NTTTDO6XW4FJVMBNONBH5WAYIC625ASPB7RUG6FMGOIA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:04:00.460134435 (589h35m0.33121747s) [upload=516842]: Next returned nil segment Dec 30 05:04:00.460180984 (589h35m0.331264029s) [upload=516842]: Waiting for error group managing segments... Dec 30 05:04:00.734126481 (589h35m0.605209526s) [noise=true , address=145.79.227.15:4876 , piece_id=TZ53FFSVWCKUIBAYPXP7JGG6MLFMR2MKKUGNAKY47NQS27HQE5ZQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.736694120 (589h35m0.607777165s) [noise=true , address=145.79.227.18:4006 , piece_id=4R2IA577WNTKXY4TXIQ77SBULCSYTO7GYJYTSIAE4C4BI5FPP7QA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.738903026 (589h35m0.609986071s) [noise=true , address=145.79.227.13:4803 , piece_id=LITYLR5VA3WFJ7UTLPRFAEQDBQALNSTSRQDYFOEMNYGPE3NEJQZA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.746488580 (589h35m0.617571625s) [noise=true , address=145.79.227.16:4911 , piece_id=2JOIF5B3NTTTDO6XW4FJVMBNONBH5WAYIC625ASPB7RUG6FMGOIA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.750122510 (589h35m0.621205555s) [noise=true , address=145.79.227.5:4222 , piece_id=ZIMAEXD7XAYAEG2NVDFTBHGDYRKKLX5WMYMKQEKIEXJLWTKFVUFA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.751076556 (589h35m0.622159602s) [noise=true , address=145.79.227.28:4726 , piece_id=OM4TIWZOREGZMUAI3EUJ2LRZ6EWA6Y6FQUXNT7DD4OUHLIM67CWA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.757339622 (589h35m0.628422667s) [noise=true , address=145.79.227.48:4301 , piece_id=PAPDGXBEA25JOVLRIXZJRRION4AEYFMXWLZDX6W7DBAYYTRI2ZOA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.766088632 (589h35m0.637171667s) [noise=true , address=145.79.227.3:4151 , piece_id=PXEEJSLYQAZZPDZ4YIOOYA2ID7M42ODUCRQUYX35AZ26LIPNFLLQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.766557344 (589h35m0.637640379s) [noise=true , address=145.79.227.38:4978 , piece_id=SZBMOYADQXARU6ZKS6CXJ4ZXLJHXGUFHUW3VRL6ND3FIUFLK2KTQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.768570367 (589h35m0.639653412s) [noise=true , address=145.79.227.52:4446 , piece_id=ZJ5HILORUROTEYZJJY3GCI4K563M6TB4VGM47KVSYU5SIWIRF6YA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.980485749 (589h35m0.851568784s) [noise=true , address=45.91.116.8:4045 , piece_id=YJ2JOLQC3CJVCXFIWJNFFZOBUUPTCHSMWZPJSMLPQDSBNO7DHGIQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:01.155105773 (589h35m1.026188829s) [noise=true , address=45.91.116.14:4189 , piece_id=3IKCTH3X6ZY3ZKWXP6RTBUZBARMKTOTXOD2QVD7XLZWVMAFMRKZA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:01.324208833 (589h35m1.195291868s) [noise=true , address=45.91.116.13:4155 , piece_id=2VEY4ZK63THG76APKXXXS6S2JO5U5WUDTGC2EKEHRMTXPNLAUS6Q, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:01.324234011 (589h35m1.195317046s) [seg_pos={0 0}, upload=516842]: Segment reached optimal threshold of 13 pieces. Dec 30 05:04:01.324468687 (589h35m1.195551732s) [noise=true , address=45.91.116.11:4117 , piece_id=AZ7ONXIOJBAAWL3K3PEUDN3EHWHYC3HAOFIT6HA4XUBIBGGV4K5Q, seg_pos={0 0}, upload=516842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52738->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52738->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52738->45.91.116.11:4117: use of closed network connection Dec 30 05:04:01.324478026 (589h35m1.195561071s) [noise=true , address=45.91.116.11:4117 , piece_id=AZ7ONXIOJBAAWL3K3PEUDN3EHWHYC3HAOFIT6HA4XUBIBGGV4K5Q, seg_pos={0 0}, upload=516842]: Long tail cancellation due to optimal threshold Dec 30 05:04:01.324760975 (589h35m1.19584404s) [noise=true , address=45.91.116.7:4009 , piece_id=XP2NS6CMVS5TEHHRRL534ZXYVYTARUZJR44BQPDG3G24YS5F4MBA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49424->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49424->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49424->45.91.116.7:4009: use of closed network connection Dec 30 05:04:01.324783016 (589h35m1.195866061s) [noise=true , address=45.91.116.7:4009 , piece_id=XP2NS6CMVS5TEHHRRL534ZXYVYTARUZJR44BQPDG3G24YS5F4MBA, seg_pos={0 0}, upload=516842]: Long tail cancellation due to optimal threshold Dec 30 05:04:01.324864472 (589h35m1.195947527s) [noise=true , address=77.90.31.10:4520 , piece_id=KDVDPUDXPJYWKITP6YH5RBGPKIFQHX2VTN5BNDOE3AJPEBTLHG4Q, seg_pos={0 0}, upload=516842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: read tcp 188.166.225.170:48146->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 05:04:01.324875703 (589h35m1.195958748s) [noise=true , address=77.90.31.10:4520 , piece_id=KDVDPUDXPJYWKITP6YH5RBGPKIFQHX2VTN5BNDOE3AJPEBTLHG4Q, seg_pos={0 0}, upload=516842]: Long tail cancellation due to optimal threshold Dec 30 05:04:01.324902514 (589h35m1.195985559s) [seg_pos={0 0}, upload=516842]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:04:01.324981614 (589h35m1.196064659s) [upload=516842]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:04:01.324986734 (589h35m1.196069769s) [upload=516842]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:04:01.615212101 (589h35m1.486295146s) [upload=516842]: Done uploading segments... Dec 30 05:04:01.615239453 (589h35m1.486322498s) [upload=516842]: Upload finished. err: <nil> Duration: 5.639293102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.....X.....X................] duration: 37.909054782s trace: https://jaeger.dev.storj.io/trace/3be231564ff45e84 FAIL (2025-12-30 05:05:09 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 5:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437041997 (589h22m14.308125042s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.437515057 (589h22m14.308598102s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.438015831 (589h22m14.309098866s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439031785 (589h22m14.310114831s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439691191 (589h22m14.310774226s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.439935637 (589h22m14.311018682s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.440948916 (589h22m14.312031951s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.441480638 (589h22m14.312563673s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.442141987 (589h22m14.313225023s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.443359656 (589h22m14.314442711s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.444147356 (589h22m14.315230391s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445113947 (589h22m14.316196992s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.445986509 (589h22m14.317069554s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:14.447400671 (589h22m14.318483706s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Uploading piece... Dec 30 04:51:15.888267947 (589h22m15.759350993s) [upload=516656]: Next returned nil segment Dec 30 04:51:15.888281523 (589h22m15.759364558s) [upload=516656]: Waiting for error group managing segments... Dec 30 04:51:16.310023137 (589h22m16.181106173s) [noise=true , address=145.79.227.33:4880 , piece_id=J5O4F5HOF7XOIETP76VQOPDLZ42POTTLLD7SZPA2GPMUMARFR6YA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.339058456 (589h22m16.210141501s) [noise=true , address=145.79.227.29:4970 , piece_id=RVIG7X7SJ2W7TBFVURDIIEUCZY3ZHFWO64SJD4L7C2MELRTVXR5Q, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.429031930 (589h22m16.300114975s) [noise=true , address=145.79.227.19:4097 , piece_id=M2YFEGJR4PMX3Z5XDHMNHTEKRL5MVLSUAEYGC3Y3Z3IS6MYRMCAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462549312 (589h22m16.333632357s) [noise=true , address=77.90.31.15:4726 , piece_id=6NXCJGCSDL6SY4TMKMV4QLSQPXXW54WGAJIJ6UNN6PFXITXAFOIQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.462617141 (589h22m16.333700196s) [noise=true , address=145.79.227.43:4188 , piece_id=HQPUJO3PRXJ72I6EI3VO7PQYH4QRXLV3MCVPEPXVUD253WXPBGVQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.475864327 (589h22m16.346947372s) [noise=true , address=77.90.31.19:4865 , piece_id=JPEVU255ZR6MSXDSRPOZZZAQZFBUV4O4JNLNJPW5FS22R24RTHFQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.512684829 (589h22m16.383767874s) [noise=true , address=145.79.227.34:4928 , piece_id=3CIM4IYTI3LW6RN45UW7R7NZWT7SAXMQITZM6RHV6APYUVOCKHJQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.575478228 (589h22m16.446561273s) [noise=true , address=77.90.31.10:4552 , piece_id=RHG2BYEBKM4ODGUXOFS5BGQ3O32RZVPTXLQYJZPLTPI6EJR4CDZA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.652204116 (589h22m16.523287161s) [noise=true , address=45.91.116.14:4217 , piece_id=VGWP5IQL476XRTITZRPFETKFPPENE55TJ53IDTOVWJGKTRCUGNKQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:16.686676327 (589h22m16.557759372s) [noise=true , address=45.91.116.8:4049 , piece_id=WTKTDAEUIT33WINGVZNWBYN25RB3PA6P4PEM6FAZBBHG4ERRO4NQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.161790743 (589h22m17.032873778s) [noise=true , address=193.149.164.143:4700 , piece_id=6I6KDNO5X6MVAWOLFTM3DJC2LULCB6D3ZZNZGCP66B4YUGCJDOEQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.168828183 (589h22m17.039911228s) [noise=true , address=145.79.227.44:4226 , piece_id=7DAMAQ5V4D4PEV66UCX4EZWLZ4DFDE6JA43I2GNFBJBNH5KYZRLA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.174630962 (589h22m17.045714007s) [noise=true , address=77.90.31.14:4695 , piece_id=674NMXYTGPOHSLHW7Q536HTB4VXAVA7IDO6LZL4O6Z4IEIAV5KFA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204056653 (589h22m17.075139698s) [noise=true , address=145.79.227.30:4783 , piece_id=HV3JYRJKDFZ2H5YVLVYTAOUFU6QMV27MF2RXUGIS6U7BHMO5KQ4A, seg_pos={0 0}, upload=516656]: Done uploading piece. err: <nil> Dec 30 04:51:17.204080248 (589h22m17.075163283s) [seg_pos={0 0}, upload=516656]: Segment reached optimal threshold of 14 pieces. Dec 30 04:51:17.204258087 (589h22m17.075341132s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36694->64.49.13.169:4707: use of closed network connection Dec 30 04:51:17.204264158 (589h22m17.075347193s) [noise=true , address=64.49.13.169:4707 , piece_id=YTPK74T4R6M4BADL3VNZUOVPQAEZUYIFQAW4SXPNSBBG2Z3AVZAA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204464951 (589h22m17.075547996s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204472745 (589h22m17.07555579s) [noise=true , address=145.79.227.40:4074 , piece_id=2KCPKOIDUXTAMBS3SWCMEGMD6JTR2FGGWQKLF3M73ETHOPSICCXQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204482615 (589h22m17.07556565s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KTMqMAjUKJ6NFK3tKvnpcKzdSxfnXTBC1hvWce28g3nP3Rhnv): protocol: expected piece hash; context canceled; EOF Dec 30 04:51:17.204488406 (589h22m17.075571451s) [noise=true , address=145.79.227.49:4354 , piece_id=2X7VENVZ6QKVCSUFUEVKAJMQUVA4AHQ3LRMKZNNAUDNGG7RUW4MA, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204646105 (589h22m17.075729151s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Nr35HEapmku71PcyhfgfDzAR9RwCuz9Q5KAEMj6TTCbrVSe7U): protocol: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43812->5.231.73.3:4015: use of closed network connection Dec 30 04:51:17.204651435 (589h22m17.075734481s) [noise=true , address=5.231.73.3:4015 , piece_id=MGQISVSATJWJ33LZGDY73XSED4FGSKY3OKCT6O44CNAHELHQQDBQ, seg_pos={0 0}, upload=516656]: Long tail cancellation due to optimal threshold Dec 30 04:51:17.204681333 (589h22m17.075764368s) [seg_pos={0 0}, upload=516656]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:51:17.204751486 (589h22m17.075834521s) [upload=516656]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:51:17.204756546 (589h22m17.075839581s) [upload=516656]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:51:17.506509713 (589h22m17.377592758s) [upload=516656]: Done uploading segments... Dec 30 04:51:17.506535523 (589h22m17.377618568s) [upload=516656]: Upload finished. err: <nil> Duration: 37.909078707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..X................X.............X...........X............X.] duration: 3.686168023s trace: https://jaeger.dev.storj.io/trace/2d91fd5cb6ac71a6 FAIL (2025-12-30 05:04:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.686192769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.X.X..X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X.] duration: 6.111594893s trace: https://jaeger.dev.storj.io/trace/68c837d837387aed FAIL (2025-12-30 05:04:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: YJ2JOLQC3CJVCXFIWJNFFZOBUUPTCHSMWZPJSMLPQDSBNO7DHGIQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 2VEY4ZK63THG76APKXXXS6S2JO5U5WUDTGC2EKEHRMTXPNLAUS6Q) context deadline exceeded ======LOGS====== : Got next segment. Inline: false Dec 30 05:03:58.486745661 (589h34m58.357828707s) [upload=516842]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:03:58.486751162 (589h34m58.357834207s) [upload=516842]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:03:58.735896835 (589h34m58.60697987s) [seg_pos={0 0}, upload=516842]: Begin upload segment... Dec 30 05:03:58.735963101 (589h34m58.607046136s) [seg_pos={0 0}, upload=516842]: Done begin upload segment. Dec 30 05:03:58.735980515 (589h34m58.60706356s) [noise=true , address=45.91.116.14:4189 , piece_id=3IKCTH3X6ZY3ZKWXP6RTBUZBARMKTOTXOD2QVD7XLZWVMAFMRKZA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.735998609 (589h34m58.607081654s) [noise=true , address=145.79.227.48:4301 , piece_id=PAPDGXBEA25JOVLRIXZJRRION4AEYFMXWLZDX6W7DBAYYTRI2ZOA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.736069213 (589h34m58.607152258s) [noise=true , address=145.79.227.38:4978 , piece_id=SZBMOYADQXARU6ZKS6CXJ4ZXLJHXGUFHUW3VRL6ND3FIUFLK2KTQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.736687250 (589h34m58.607770295s) [noise=true , address=145.79.227.18:4006 , piece_id=4R2IA577WNTKXY4TXIQ77SBULCSYTO7GYJYTSIAE4C4BI5FPP7QA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.737170360 (589h34m58.608253405s) [noise=true , address=145.79.227.3:4151 , piece_id=PXEEJSLYQAZZPDZ4YIOOYA2ID7M42ODUCRQUYX35AZ26LIPNFLLQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.737573177 (589h34m58.608656212s) [noise=true , address=145.79.227.15:4876 , piece_id=TZ53FFSVWCKUIBAYPXP7JGG6MLFMR2MKKUGNAKY47NQS27HQE5ZQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.738027201 (589h34m58.609110246s) [noise=true , address=45.91.116.11:4117 , piece_id=AZ7ONXIOJBAAWL3K3PEUDN3EHWHYC3HAOFIT6HA4XUBIBGGV4K5Q, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.738543905 (589h34m58.60962694s) [noise=true , address=145.79.227.28:4726 , piece_id=OM4TIWZOREGZMUAI3EUJ2LRZ6EWA6Y6FQUXNT7DD4OUHLIM67CWA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.739062273 (589h34m58.610145318s) [noise=true , address=45.91.116.13:4155 , piece_id=2VEY4ZK63THG76APKXXXS6S2JO5U5WUDTGC2EKEHRMTXPNLAUS6Q, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.739626457 (589h34m58.610709502s) [noise=true , address=145.79.227.5:4222 , piece_id=ZIMAEXD7XAYAEG2NVDFTBHGDYRKKLX5WMYMKQEKIEXJLWTKFVUFA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.747476504 (589h34m58.618559549s) [noise=true , address=145.79.227.52:4446 , piece_id=ZJ5HILORUROTEYZJJY3GCI4K563M6TB4VGM47KVSYU5SIWIRF6YA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.748958116 (589h34m58.620041151s) [noise=true , address=145.79.227.13:4803 , piece_id=LITYLR5VA3WFJ7UTLPRFAEQDBQALNSTSRQDYFOEMNYGPE3NEJQZA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.751521547 (589h34m58.622604592s) [noise=true , address=45.91.116.8:4045 , piece_id=YJ2JOLQC3CJVCXFIWJNFFZOBUUPTCHSMWZPJSMLPQDSBNO7DHGIQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.754403955 (589h34m58.62548699s) [noise=true , address=45.91.116.7:4009 , piece_id=XP2NS6CMVS5TEHHRRL534ZXYVYTARUZJR44BQPDG3G24YS5F4MBA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.755931414 (589h34m58.627014449s) [noise=true , address=77.90.31.10:4520 , piece_id=KDVDPUDXPJYWKITP6YH5RBGPKIFQHX2VTN5BNDOE3AJPEBTLHG4Q, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.756599896 (589h34m58.627682932s) [noise=true , address=145.79.227.16:4911 , piece_id=2JOIF5B3NTTTDO6XW4FJVMBNONBH5WAYIC625ASPB7RUG6FMGOIA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:04:00.460134435 (589h35m0.33121747s) [upload=516842]: Next returned nil segment Dec 30 05:04:00.460180984 (589h35m0.331264029s) [upload=516842]: Waiting for error group managing segments... Dec 30 05:04:00.734126481 (589h35m0.605209526s) [noise=true , address=145.79.227.15:4876 , piece_id=TZ53FFSVWCKUIBAYPXP7JGG6MLFMR2MKKUGNAKY47NQS27HQE5ZQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.736694120 (589h35m0.607777165s) [noise=true , address=145.79.227.18:4006 , piece_id=4R2IA577WNTKXY4TXIQ77SBULCSYTO7GYJYTSIAE4C4BI5FPP7QA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.738903026 (589h35m0.609986071s) [noise=true , address=145.79.227.13:4803 , piece_id=LITYLR5VA3WFJ7UTLPRFAEQDBQALNSTSRQDYFOEMNYGPE3NEJQZA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.746488580 (589h35m0.617571625s) [noise=true , address=145.79.227.16:4911 , piece_id=2JOIF5B3NTTTDO6XW4FJVMBNONBH5WAYIC625ASPB7RUG6FMGOIA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.750122510 (589h35m0.621205555s) [noise=true , address=145.79.227.5:4222 , piece_id=ZIMAEXD7XAYAEG2NVDFTBHGDYRKKLX5WMYMKQEKIEXJLWTKFVUFA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.751076556 (589h35m0.622159602s) [noise=true , address=145.79.227.28:4726 , piece_id=OM4TIWZOREGZMUAI3EUJ2LRZ6EWA6Y6FQUXNT7DD4OUHLIM67CWA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.757339622 (589h35m0.628422667s) [noise=true , address=145.79.227.48:4301 , piece_id=PAPDGXBEA25JOVLRIXZJRRION4AEYFMXWLZDX6W7DBAYYTRI2ZOA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.766088632 (589h35m0.637171667s) [noise=true , address=145.79.227.3:4151 , piece_id=PXEEJSLYQAZZPDZ4YIOOYA2ID7M42ODUCRQUYX35AZ26LIPNFLLQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.766557344 (589h35m0.637640379s) [noise=true , address=145.79.227.38:4978 , piece_id=SZBMOYADQXARU6ZKS6CXJ4ZXLJHXGUFHUW3VRL6ND3FIUFLK2KTQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.768570367 (589h35m0.639653412s) [noise=true , address=145.79.227.52:4446 , piece_id=ZJ5HILORUROTEYZJJY3GCI4K563M6TB4VGM47KVSYU5SIWIRF6YA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.980485749 (589h35m0.851568784s) [noise=true , address=45.91.116.8:4045 , piece_id=YJ2JOLQC3CJVCXFIWJNFFZOBUUPTCHSMWZPJSMLPQDSBNO7DHGIQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:01.155105773 (589h35m1.026188829s) [noise=true , address=45.91.116.14:4189 , piece_id=3IKCTH3X6ZY3ZKWXP6RTBUZBARMKTOTXOD2QVD7XLZWVMAFMRKZA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:01.324208833 (589h35m1.195291868s) [noise=true , address=45.91.116.13:4155 , piece_id=2VEY4ZK63THG76APKXXXS6S2JO5U5WUDTGC2EKEHRMTXPNLAUS6Q, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:01.324234011 (589h35m1.195317046s) [seg_pos={0 0}, upload=516842]: Segment reached optimal threshold of 13 pieces. Dec 30 05:04:01.324468687 (589h35m1.195551732s) [noise=true , address=45.91.116.11:4117 , piece_id=AZ7ONXIOJBAAWL3K3PEUDN3EHWHYC3HAOFIT6HA4XUBIBGGV4K5Q, seg_pos={0 0}, upload=516842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52738->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52738->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52738->45.91.116.11:4117: use of closed network connection Dec 30 05:04:01.324478026 (589h35m1.195561071s) [noise=true , address=45.91.116.11:4117 , piece_id=AZ7ONXIOJBAAWL3K3PEUDN3EHWHYC3HAOFIT6HA4XUBIBGGV4K5Q, seg_pos={0 0}, upload=516842]: Long tail cancellation due to optimal threshold Dec 30 05:04:01.324760975 (589h35m1.19584404s) [noise=true , address=45.91.116.7:4009 , piece_id=XP2NS6CMVS5TEHHRRL534ZXYVYTARUZJR44BQPDG3G24YS5F4MBA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49424->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49424->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49424->45.91.116.7:4009: use of closed network connection Dec 30 05:04:01.324783016 (589h35m1.195866061s) [noise=true , address=45.91.116.7:4009 , piece_id=XP2NS6CMVS5TEHHRRL534ZXYVYTARUZJR44BQPDG3G24YS5F4MBA, seg_pos={0 0}, upload=516842]: Long tail cancellation due to optimal threshold Dec 30 05:04:01.324864472 (589h35m1.195947527s) [noise=true , address=77.90.31.10:4520 , piece_id=KDVDPUDXPJYWKITP6YH5RBGPKIFQHX2VTN5BNDOE3AJPEBTLHG4Q, seg_pos={0 0}, upload=516842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: read tcp 188.166.225.170:48146->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 05:04:01.324875703 (589h35m1.195958748s) [noise=true , address=77.90.31.10:4520 , piece_id=KDVDPUDXPJYWKITP6YH5RBGPKIFQHX2VTN5BNDOE3AJPEBTLHG4Q, seg_pos={0 0}, upload=516842]: Long tail cancellation due to optimal threshold Dec 30 05:04:01.324902514 (589h35m1.195985559s) [seg_pos={0 0}, upload=516842]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:04:01.324981614 (589h35m1.196064659s) [upload=516842]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:04:01.324986734 (589h35m1.196069769s) [upload=516842]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:04:01.615212101 (589h35m1.486295146s) [upload=516842]: Done uploading segments... Dec 30 05:04:01.615239453 (589h35m1.486322498s) [upload=516842]: Upload finished. err: <nil> Duration: 6.111624768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X.X.X..X....XX...XXX...X....X.X.....X..XX.X....X.X....X.X] duration: 30.004622252s trace: https://jaeger.dev.storj.io/trace/68c837d837387aed FAIL (2025-12-30 05:04:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: YJ2JOLQC3CJVCXFIWJNFFZOBUUPTCHSMWZPJSMLPQDSBNO7DHGIQ) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 2VEY4ZK63THG76APKXXXS6S2JO5U5WUDTGC2EKEHRMTXPNLAUS6Q) context deadline exceeded ======LOGS====== : Got next segment. Inline: false Dec 30 05:03:58.486745661 (589h34m58.357828707s) [upload=516842]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:03:58.486751162 (589h34m58.357834207s) [upload=516842]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:03:58.735896835 (589h34m58.60697987s) [seg_pos={0 0}, upload=516842]: Begin upload segment... Dec 30 05:03:58.735963101 (589h34m58.607046136s) [seg_pos={0 0}, upload=516842]: Done begin upload segment. Dec 30 05:03:58.735980515 (589h34m58.60706356s) [noise=true , address=45.91.116.14:4189 , piece_id=3IKCTH3X6ZY3ZKWXP6RTBUZBARMKTOTXOD2QVD7XLZWVMAFMRKZA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.735998609 (589h34m58.607081654s) [noise=true , address=145.79.227.48:4301 , piece_id=PAPDGXBEA25JOVLRIXZJRRION4AEYFMXWLZDX6W7DBAYYTRI2ZOA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.736069213 (589h34m58.607152258s) [noise=true , address=145.79.227.38:4978 , piece_id=SZBMOYADQXARU6ZKS6CXJ4ZXLJHXGUFHUW3VRL6ND3FIUFLK2KTQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.736687250 (589h34m58.607770295s) [noise=true , address=145.79.227.18:4006 , piece_id=4R2IA577WNTKXY4TXIQ77SBULCSYTO7GYJYTSIAE4C4BI5FPP7QA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.737170360 (589h34m58.608253405s) [noise=true , address=145.79.227.3:4151 , piece_id=PXEEJSLYQAZZPDZ4YIOOYA2ID7M42ODUCRQUYX35AZ26LIPNFLLQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.737573177 (589h34m58.608656212s) [noise=true , address=145.79.227.15:4876 , piece_id=TZ53FFSVWCKUIBAYPXP7JGG6MLFMR2MKKUGNAKY47NQS27HQE5ZQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.738027201 (589h34m58.609110246s) [noise=true , address=45.91.116.11:4117 , piece_id=AZ7ONXIOJBAAWL3K3PEUDN3EHWHYC3HAOFIT6HA4XUBIBGGV4K5Q, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.738543905 (589h34m58.60962694s) [noise=true , address=145.79.227.28:4726 , piece_id=OM4TIWZOREGZMUAI3EUJ2LRZ6EWA6Y6FQUXNT7DD4OUHLIM67CWA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.739062273 (589h34m58.610145318s) [noise=true , address=45.91.116.13:4155 , piece_id=2VEY4ZK63THG76APKXXXS6S2JO5U5WUDTGC2EKEHRMTXPNLAUS6Q, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.739626457 (589h34m58.610709502s) [noise=true , address=145.79.227.5:4222 , piece_id=ZIMAEXD7XAYAEG2NVDFTBHGDYRKKLX5WMYMKQEKIEXJLWTKFVUFA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.747476504 (589h34m58.618559549s) [noise=true , address=145.79.227.52:4446 , piece_id=ZJ5HILORUROTEYZJJY3GCI4K563M6TB4VGM47KVSYU5SIWIRF6YA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.748958116 (589h34m58.620041151s) [noise=true , address=145.79.227.13:4803 , piece_id=LITYLR5VA3WFJ7UTLPRFAEQDBQALNSTSRQDYFOEMNYGPE3NEJQZA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.751521547 (589h34m58.622604592s) [noise=true , address=45.91.116.8:4045 , piece_id=YJ2JOLQC3CJVCXFIWJNFFZOBUUPTCHSMWZPJSMLPQDSBNO7DHGIQ, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.754403955 (589h34m58.62548699s) [noise=true , address=45.91.116.7:4009 , piece_id=XP2NS6CMVS5TEHHRRL534ZXYVYTARUZJR44BQPDG3G24YS5F4MBA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.755931414 (589h34m58.627014449s) [noise=true , address=77.90.31.10:4520 , piece_id=KDVDPUDXPJYWKITP6YH5RBGPKIFQHX2VTN5BNDOE3AJPEBTLHG4Q, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:03:58.756599896 (589h34m58.627682932s) [noise=true , address=145.79.227.16:4911 , piece_id=2JOIF5B3NTTTDO6XW4FJVMBNONBH5WAYIC625ASPB7RUG6FMGOIA, seg_pos={0 0}, upload=516842]: Uploading piece... Dec 30 05:04:00.460134435 (589h35m0.33121747s) [upload=516842]: Next returned nil segment Dec 30 05:04:00.460180984 (589h35m0.331264029s) [upload=516842]: Waiting for error group managing segments... Dec 30 05:04:00.734126481 (589h35m0.605209526s) [noise=true , address=145.79.227.15:4876 , piece_id=TZ53FFSVWCKUIBAYPXP7JGG6MLFMR2MKKUGNAKY47NQS27HQE5ZQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.736694120 (589h35m0.607777165s) [noise=true , address=145.79.227.18:4006 , piece_id=4R2IA577WNTKXY4TXIQ77SBULCSYTO7GYJYTSIAE4C4BI5FPP7QA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.738903026 (589h35m0.609986071s) [noise=true , address=145.79.227.13:4803 , piece_id=LITYLR5VA3WFJ7UTLPRFAEQDBQALNSTSRQDYFOEMNYGPE3NEJQZA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.746488580 (589h35m0.617571625s) [noise=true , address=145.79.227.16:4911 , piece_id=2JOIF5B3NTTTDO6XW4FJVMBNONBH5WAYIC625ASPB7RUG6FMGOIA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.750122510 (589h35m0.621205555s) [noise=true , address=145.79.227.5:4222 , piece_id=ZIMAEXD7XAYAEG2NVDFTBHGDYRKKLX5WMYMKQEKIEXJLWTKFVUFA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.751076556 (589h35m0.622159602s) [noise=true , address=145.79.227.28:4726 , piece_id=OM4TIWZOREGZMUAI3EUJ2LRZ6EWA6Y6FQUXNT7DD4OUHLIM67CWA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.757339622 (589h35m0.628422667s) [noise=true , address=145.79.227.48:4301 , piece_id=PAPDGXBEA25JOVLRIXZJRRION4AEYFMXWLZDX6W7DBAYYTRI2ZOA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.766088632 (589h35m0.637171667s) [noise=true , address=145.79.227.3:4151 , piece_id=PXEEJSLYQAZZPDZ4YIOOYA2ID7M42ODUCRQUYX35AZ26LIPNFLLQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.766557344 (589h35m0.637640379s) [noise=true , address=145.79.227.38:4978 , piece_id=SZBMOYADQXARU6ZKS6CXJ4ZXLJHXGUFHUW3VRL6ND3FIUFLK2KTQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.768570367 (589h35m0.639653412s) [noise=true , address=145.79.227.52:4446 , piece_id=ZJ5HILORUROTEYZJJY3GCI4K563M6TB4VGM47KVSYU5SIWIRF6YA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:00.980485749 (589h35m0.851568784s) [noise=true , address=45.91.116.8:4045 , piece_id=YJ2JOLQC3CJVCXFIWJNFFZOBUUPTCHSMWZPJSMLPQDSBNO7DHGIQ, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:01.155105773 (589h35m1.026188829s) [noise=true , address=45.91.116.14:4189 , piece_id=3IKCTH3X6ZY3ZKWXP6RTBUZBARMKTOTXOD2QVD7XLZWVMAFMRKZA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:01.324208833 (589h35m1.195291868s) [noise=true , address=45.91.116.13:4155 , piece_id=2VEY4ZK63THG76APKXXXS6S2JO5U5WUDTGC2EKEHRMTXPNLAUS6Q, seg_pos={0 0}, upload=516842]: Done uploading piece. err: <nil> Dec 30 05:04:01.324234011 (589h35m1.195317046s) [seg_pos={0 0}, upload=516842]: Segment reached optimal threshold of 13 pieces. Dec 30 05:04:01.324468687 (589h35m1.195551732s) [noise=true , address=45.91.116.11:4117 , piece_id=AZ7ONXIOJBAAWL3K3PEUDN3EHWHYC3HAOFIT6HA4XUBIBGGV4K5Q, seg_pos={0 0}, upload=516842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:52738->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:52738->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52738->45.91.116.11:4117: use of closed network connection Dec 30 05:04:01.324478026 (589h35m1.195561071s) [noise=true , address=45.91.116.11:4117 , piece_id=AZ7ONXIOJBAAWL3K3PEUDN3EHWHYC3HAOFIT6HA4XUBIBGGV4K5Q, seg_pos={0 0}, upload=516842]: Long tail cancellation due to optimal threshold Dec 30 05:04:01.324760975 (589h35m1.19584404s) [noise=true , address=45.91.116.7:4009 , piece_id=XP2NS6CMVS5TEHHRRL534ZXYVYTARUZJR44BQPDG3G24YS5F4MBA, seg_pos={0 0}, upload=516842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:49424->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:49424->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49424->45.91.116.7:4009: use of closed network connection Dec 30 05:04:01.324783016 (589h35m1.195866061s) [noise=true , address=45.91.116.7:4009 , piece_id=XP2NS6CMVS5TEHHRRL534ZXYVYTARUZJR44BQPDG3G24YS5F4MBA, seg_pos={0 0}, upload=516842]: Long tail cancellation due to optimal threshold Dec 30 05:04:01.324864472 (589h35m1.195947527s) [noise=true , address=77.90.31.10:4520 , piece_id=KDVDPUDXPJYWKITP6YH5RBGPKIFQHX2VTN5BNDOE3AJPEBTLHG4Q, seg_pos={0 0}, upload=516842]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: read tcp 188.166.225.170:48146->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 05:04:01.324875703 (589h35m1.195958748s) [noise=true , address=77.90.31.10:4520 , piece_id=KDVDPUDXPJYWKITP6YH5RBGPKIFQHX2VTN5BNDOE3AJPEBTLHG4Q, seg_pos={0 0}, upload=516842]: Long tail cancellation due to optimal threshold Dec 30 05:04:01.324902514 (589h35m1.195985559s) [seg_pos={0 0}, upload=516842]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:04:01.324981614 (589h35m1.196064659s) [upload=516842]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:04:01.324986734 (589h35m1.196069769s) [upload=516842]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:04:01.615212101 (589h35m1.486295146s) [upload=516842]: Done uploading segments... Dec 30 05:04:01.615239453 (589h35m1.486322498s) [upload=516842]: Upload finished. err: <nil> Duration: 30.004646789s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)