Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X............X.............................................] duration: 28.567884177s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:37:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 28.567910386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [XX............XX..X.....X..............X................X...] duration: 3.622674943s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:37:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.622701934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX...] duration: 5.943484903s trace: https://jaeger.dev.storj.io/trace/5d491286618aa306 FAIL (2025-12-30 04:37:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ) context deadline exceeded ======LOGS====== Dec 30 04:33:53.183104152 (589h4m53.054187197s) [upload=516409]: Starting upload Dec 30 04:33:53.183214783 (589h4m53.054297828s) [upload=516409]: Done starting upload Dec 30 04:33:53.183247104 (589h4m53.054330149s) [upload=516409]: Uploading segments... Dec 30 04:33:53.183636365 (589h4m53.0547194s) [upload=516409]: Got next segment. Inline: false Dec 30 04:33:53.183695148 (589h4m53.054778193s) [upload=516409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:33:53.183718471 (589h4m53.054801506s) [upload=516409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:33:53.430021290 (589h4m53.301104355s) [seg_pos={0 0}, upload=516409]: Begin upload segment... Dec 30 04:33:53.430115249 (589h4m53.301198294s) [seg_pos={0 0}, upload=516409]: Done begin upload segment. Dec 30 04:33:53.430177378 (589h4m53.301260413s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.430225990 (589h4m53.301309035s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.431398643 (589h4m53.302481688s) [noise=true , address=77.90.31.14:4661 , piece_id=Y2MZVSCLBQFWZ7RHRQ6ZPYHOZCJ5HY7MZ6KP4LYOKZBJRS5EZASQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.434681364 (589h4m53.305764459s) [noise=true , address=145.79.227.35:4482 , piece_id=O4JKIDKYF2X3FDSCOQ5YFWZC6R4X3BKDPD6IOQ7GFBGLLVNIZYKQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.435664345 (589h4m53.30674739s) [noise=true , address=145.79.227.14:4837 , piece_id=T2N3GRL4RHBQW7RXS463IFLBKVOJASW66O3SKZOF4LORHPCUBHOQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.436612571 (589h4m53.307695616s) [noise=true , address=145.79.227.15:4873 , piece_id=H6V2GHW6WGYPSVZONKK63MP6T764OJXHMU53R3P4EMH6MWOEMNDA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.437477207 (589h4m53.308560252s) [noise=true , address=145.79.227.17:4972 , piece_id=6JB5VMQEIZMOGEULWAESCORSCPSAFVPMO4ZQ4A2KURERRUIDW7FA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.440643055 (589h4m53.311726101s) [noise=true , address=145.79.227.24:4582 , piece_id=2TZB3WZWPCWUXW36Z3JGCTFECKNC5OJPLBKLHB3U2VNEGYIYFKCA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.444232941 (589h4m53.315315986s) [noise=true , address=77.90.31.10:4518 , piece_id=2YFBS7RXT2LCS6T5JKXELD6XUZ3JJUJRGTIE3A6BKVUR6RUTGQJA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.447040297 (589h4m53.318123342s) [noise=true , address=145.79.227.3:4149 , piece_id=J4KTQR5BCQPFLZUVCPS3U6WNRJB47YGA5U5MVZJ5Y3RXTLTW5WPA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.449070974 (589h4m53.320154019s) [noise=true , address=77.90.31.12:4590 , piece_id=IOAXSQ42ROZT6V6I3V7L4E6SASHM7MH4GLSHR2BINBSBPV43QVJA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.450860581 (589h4m53.321943626s) [noise=true , address=77.90.31.11:4555 , piece_id=DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.452555569 (589h4m53.323638614s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.456353120 (589h4m53.327436165s) [noise=true , address=45.91.116.12:4226 , piece_id=O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.460079255 (589h4m53.3311623s) [noise=true , address=45.91.116.7:4009 , piece_id=YJBCELXP2YIZIBIYL3EV3XQ45FTVXRSLLJHPKHNQYMSOVDZ2MGOQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.464950641 (589h4m53.336033686s) [noise=true , address=145.79.227.2:4113 , piece_id=22AA2HEJ237XEOBQAHJ26N5J4FSNGP6HXMY5LM22FY44VRGKPHGQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:55.139659825 (589h4m55.010742871s) [upload=516409]: Next returned nil segment Dec 30 04:33:55.139680214 (589h4m55.010763259s) [upload=516409]: Waiting for error group managing segments... Dec 30 04:33:55.419515103 (589h4m55.290598148s) [noise=true , address=145.79.227.17:4972 , piece_id=6JB5VMQEIZMOGEULWAESCORSCPSAFVPMO4ZQ4A2KURERRUIDW7FA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.432174058 (589h4m55.303257093s) [noise=true , address=145.79.227.15:4873 , piece_id=H6V2GHW6WGYPSVZONKK63MP6T764OJXHMU53R3P4EMH6MWOEMNDA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.437302654 (589h4m55.308385689s) [noise=true , address=145.79.227.14:4837 , piece_id=T2N3GRL4RHBQW7RXS463IFLBKVOJASW66O3SKZOF4LORHPCUBHOQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.437565775 (589h4m55.30864882s) [noise=true , address=145.79.227.24:4582 , piece_id=2TZB3WZWPCWUXW36Z3JGCTFECKNC5OJPLBKLHB3U2VNEGYIYFKCA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.457133444 (589h4m55.328216489s) [noise=true , address=145.79.227.2:4113 , piece_id=22AA2HEJ237XEOBQAHJ26N5J4FSNGP6HXMY5LM22FY44VRGKPHGQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.457756110 (589h4m55.328839155s) [noise=true , address=145.79.227.3:4149 , piece_id=J4KTQR5BCQPFLZUVCPS3U6WNRJB47YGA5U5MVZJ5Y3RXTLTW5WPA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.604637414 (589h4m55.475720459s) [noise=true , address=145.79.227.35:4482 , piece_id=O4JKIDKYF2X3FDSCOQ5YFWZC6R4X3BKDPD6IOQ7GFBGLLVNIZYKQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.646242637 (589h4m55.517325672s) [noise=true , address=77.90.31.12:4590 , piece_id=IOAXSQ42ROZT6V6I3V7L4E6SASHM7MH4GLSHR2BINBSBPV43QVJA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.716537068 (589h4m55.587620113s) [noise=true , address=77.90.31.10:4518 , piece_id=2YFBS7RXT2LCS6T5JKXELD6XUZ3JJUJRGTIE3A6BKVUR6RUTGQJA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.843169556 (589h4m55.714252601s) [noise=true , address=45.91.116.7:4009 , piece_id=YJBCELXP2YIZIBIYL3EV3XQ45FTVXRSLLJHPKHNQYMSOVDZ2MGOQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.224883287 (589h4m56.095966332s) [noise=true , address=77.90.31.11:4555 , piece_id=DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.489291652 (589h4m56.360374707s) [noise=true , address=45.91.116.12:4226 , piece_id=O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.594859078 (589h4m56.465942124s) [noise=true , address=77.90.31.14:4661 , piece_id=Y2MZVSCLBQFWZ7RHRQ6ZPYHOZCJ5HY7MZ6KP4LYOKZBJRS5EZASQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.594881130 (589h4m56.465964175s) [seg_pos={0 0}, upload=516409]: Segment reached optimal threshold of 13 pieces. Dec 30 04:33:56.595107762 (589h4m56.466190797s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 04:33:56.595115196 (589h4m56.466198241s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595130866 (589h4m56.466213911s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 04:33:56.595144041 (589h4m56.466227117s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595385601 (589h4m56.466468636s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection Dec 30 04:33:56.595395931 (589h4m56.466478976s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595419927 (589h4m56.466502972s) [seg_pos={0 0}, upload=516409]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:33:56.595511090 (589h4m56.466594135s) [upload=516409]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:33:56.595517592 (589h4m56.466600637s) [upload=516409]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:33:56.891943372 (589h4m56.763026408s) [upload=516409]: Done uploading segments... Dec 30 04:33:56.891964744 (589h4m56.763047789s) [upload=516409]: Upload finished. err: <nil> Duration: 5.943514609s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X............X............................................] duration: 15.423104976s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:36:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 15.423134222s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [XXX............XX..X.....X..............X................X..] duration: 5.000943407s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:36:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001027656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X............X...........................................] duration: 43.770322356s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:36:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 43.770350229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX..] duration: 6.099202247s trace: https://jaeger.dev.storj.io/trace/5d491286618aa306 FAIL (2025-12-30 04:35:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ) context deadline exceeded ======LOGS====== Dec 30 04:33:53.183104152 (589h4m53.054187197s) [upload=516409]: Starting upload Dec 30 04:33:53.183214783 (589h4m53.054297828s) [upload=516409]: Done starting upload Dec 30 04:33:53.183247104 (589h4m53.054330149s) [upload=516409]: Uploading segments... Dec 30 04:33:53.183636365 (589h4m53.0547194s) [upload=516409]: Got next segment. Inline: false Dec 30 04:33:53.183695148 (589h4m53.054778193s) [upload=516409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:33:53.183718471 (589h4m53.054801506s) [upload=516409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:33:53.430021290 (589h4m53.301104355s) [seg_pos={0 0}, upload=516409]: Begin upload segment... Dec 30 04:33:53.430115249 (589h4m53.301198294s) [seg_pos={0 0}, upload=516409]: Done begin upload segment. Dec 30 04:33:53.430177378 (589h4m53.301260413s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.430225990 (589h4m53.301309035s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.431398643 (589h4m53.302481688s) [noise=true , address=77.90.31.14:4661 , piece_id=Y2MZVSCLBQFWZ7RHRQ6ZPYHOZCJ5HY7MZ6KP4LYOKZBJRS5EZASQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.434681364 (589h4m53.305764459s) [noise=true , address=145.79.227.35:4482 , piece_id=O4JKIDKYF2X3FDSCOQ5YFWZC6R4X3BKDPD6IOQ7GFBGLLVNIZYKQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.435664345 (589h4m53.30674739s) [noise=true , address=145.79.227.14:4837 , piece_id=T2N3GRL4RHBQW7RXS463IFLBKVOJASW66O3SKZOF4LORHPCUBHOQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.436612571 (589h4m53.307695616s) [noise=true , address=145.79.227.15:4873 , piece_id=H6V2GHW6WGYPSVZONKK63MP6T764OJXHMU53R3P4EMH6MWOEMNDA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.437477207 (589h4m53.308560252s) [noise=true , address=145.79.227.17:4972 , piece_id=6JB5VMQEIZMOGEULWAESCORSCPSAFVPMO4ZQ4A2KURERRUIDW7FA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.440643055 (589h4m53.311726101s) [noise=true , address=145.79.227.24:4582 , piece_id=2TZB3WZWPCWUXW36Z3JGCTFECKNC5OJPLBKLHB3U2VNEGYIYFKCA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.444232941 (589h4m53.315315986s) [noise=true , address=77.90.31.10:4518 , piece_id=2YFBS7RXT2LCS6T5JKXELD6XUZ3JJUJRGTIE3A6BKVUR6RUTGQJA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.447040297 (589h4m53.318123342s) [noise=true , address=145.79.227.3:4149 , piece_id=J4KTQR5BCQPFLZUVCPS3U6WNRJB47YGA5U5MVZJ5Y3RXTLTW5WPA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.449070974 (589h4m53.320154019s) [noise=true , address=77.90.31.12:4590 , piece_id=IOAXSQ42ROZT6V6I3V7L4E6SASHM7MH4GLSHR2BINBSBPV43QVJA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.450860581 (589h4m53.321943626s) [noise=true , address=77.90.31.11:4555 , piece_id=DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.452555569 (589h4m53.323638614s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.456353120 (589h4m53.327436165s) [noise=true , address=45.91.116.12:4226 , piece_id=O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.460079255 (589h4m53.3311623s) [noise=true , address=45.91.116.7:4009 , piece_id=YJBCELXP2YIZIBIYL3EV3XQ45FTVXRSLLJHPKHNQYMSOVDZ2MGOQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.464950641 (589h4m53.336033686s) [noise=true , address=145.79.227.2:4113 , piece_id=22AA2HEJ237XEOBQAHJ26N5J4FSNGP6HXMY5LM22FY44VRGKPHGQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:55.139659825 (589h4m55.010742871s) [upload=516409]: Next returned nil segment Dec 30 04:33:55.139680214 (589h4m55.010763259s) [upload=516409]: Waiting for error group managing segments... Dec 30 04:33:55.419515103 (589h4m55.290598148s) [noise=true , address=145.79.227.17:4972 , piece_id=6JB5VMQEIZMOGEULWAESCORSCPSAFVPMO4ZQ4A2KURERRUIDW7FA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.432174058 (589h4m55.303257093s) [noise=true , address=145.79.227.15:4873 , piece_id=H6V2GHW6WGYPSVZONKK63MP6T764OJXHMU53R3P4EMH6MWOEMNDA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.437302654 (589h4m55.308385689s) [noise=true , address=145.79.227.14:4837 , piece_id=T2N3GRL4RHBQW7RXS463IFLBKVOJASW66O3SKZOF4LORHPCUBHOQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.437565775 (589h4m55.30864882s) [noise=true , address=145.79.227.24:4582 , piece_id=2TZB3WZWPCWUXW36Z3JGCTFECKNC5OJPLBKLHB3U2VNEGYIYFKCA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.457133444 (589h4m55.328216489s) [noise=true , address=145.79.227.2:4113 , piece_id=22AA2HEJ237XEOBQAHJ26N5J4FSNGP6HXMY5LM22FY44VRGKPHGQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.457756110 (589h4m55.328839155s) [noise=true , address=145.79.227.3:4149 , piece_id=J4KTQR5BCQPFLZUVCPS3U6WNRJB47YGA5U5MVZJ5Y3RXTLTW5WPA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.604637414 (589h4m55.475720459s) [noise=true , address=145.79.227.35:4482 , piece_id=O4JKIDKYF2X3FDSCOQ5YFWZC6R4X3BKDPD6IOQ7GFBGLLVNIZYKQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.646242637 (589h4m55.517325672s) [noise=true , address=77.90.31.12:4590 , piece_id=IOAXSQ42ROZT6V6I3V7L4E6SASHM7MH4GLSHR2BINBSBPV43QVJA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.716537068 (589h4m55.587620113s) [noise=true , address=77.90.31.10:4518 , piece_id=2YFBS7RXT2LCS6T5JKXELD6XUZ3JJUJRGTIE3A6BKVUR6RUTGQJA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.843169556 (589h4m55.714252601s) [noise=true , address=45.91.116.7:4009 , piece_id=YJBCELXP2YIZIBIYL3EV3XQ45FTVXRSLLJHPKHNQYMSOVDZ2MGOQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.224883287 (589h4m56.095966332s) [noise=true , address=77.90.31.11:4555 , piece_id=DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.489291652 (589h4m56.360374707s) [noise=true , address=45.91.116.12:4226 , piece_id=O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.594859078 (589h4m56.465942124s) [noise=true , address=77.90.31.14:4661 , piece_id=Y2MZVSCLBQFWZ7RHRQ6ZPYHOZCJ5HY7MZ6KP4LYOKZBJRS5EZASQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.594881130 (589h4m56.465964175s) [seg_pos={0 0}, upload=516409]: Segment reached optimal threshold of 13 pieces. Dec 30 04:33:56.595107762 (589h4m56.466190797s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 04:33:56.595115196 (589h4m56.466198241s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595130866 (589h4m56.466213911s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 04:33:56.595144041 (589h4m56.466227117s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595385601 (589h4m56.466468636s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection Dec 30 04:33:56.595395931 (589h4m56.466478976s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595419927 (589h4m56.466502972s) [seg_pos={0 0}, upload=516409]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:33:56.595511090 (589h4m56.466594135s) [upload=516409]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:33:56.595517592 (589h4m56.466600637s) [upload=516409]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:33:56.891943372 (589h4m56.763026408s) [upload=516409]: Done uploading segments... Dec 30 04:33:56.891964744 (589h4m56.763047789s) [upload=516409]: Upload finished. err: <nil> Duration: 6.099231744s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.XXX............XX..X.....X..............X................X.] duration: 3.964654613s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:35:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.964672036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX.] duration: 28.093796491s trace: https://jaeger.dev.storj.io/trace/5d491286618aa306 FAIL (2025-12-30 04:35:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ) context deadline exceeded ======LOGS====== Dec 30 04:33:53.183104152 (589h4m53.054187197s) [upload=516409]: Starting upload Dec 30 04:33:53.183214783 (589h4m53.054297828s) [upload=516409]: Done starting upload Dec 30 04:33:53.183247104 (589h4m53.054330149s) [upload=516409]: Uploading segments... Dec 30 04:33:53.183636365 (589h4m53.0547194s) [upload=516409]: Got next segment. Inline: false Dec 30 04:33:53.183695148 (589h4m53.054778193s) [upload=516409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:33:53.183718471 (589h4m53.054801506s) [upload=516409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:33:53.430021290 (589h4m53.301104355s) [seg_pos={0 0}, upload=516409]: Begin upload segment... Dec 30 04:33:53.430115249 (589h4m53.301198294s) [seg_pos={0 0}, upload=516409]: Done begin upload segment. Dec 30 04:33:53.430177378 (589h4m53.301260413s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.430225990 (589h4m53.301309035s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.431398643 (589h4m53.302481688s) [noise=true , address=77.90.31.14:4661 , piece_id=Y2MZVSCLBQFWZ7RHRQ6ZPYHOZCJ5HY7MZ6KP4LYOKZBJRS5EZASQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.434681364 (589h4m53.305764459s) [noise=true , address=145.79.227.35:4482 , piece_id=O4JKIDKYF2X3FDSCOQ5YFWZC6R4X3BKDPD6IOQ7GFBGLLVNIZYKQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.435664345 (589h4m53.30674739s) [noise=true , address=145.79.227.14:4837 , piece_id=T2N3GRL4RHBQW7RXS463IFLBKVOJASW66O3SKZOF4LORHPCUBHOQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.436612571 (589h4m53.307695616s) [noise=true , address=145.79.227.15:4873 , piece_id=H6V2GHW6WGYPSVZONKK63MP6T764OJXHMU53R3P4EMH6MWOEMNDA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.437477207 (589h4m53.308560252s) [noise=true , address=145.79.227.17:4972 , piece_id=6JB5VMQEIZMOGEULWAESCORSCPSAFVPMO4ZQ4A2KURERRUIDW7FA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.440643055 (589h4m53.311726101s) [noise=true , address=145.79.227.24:4582 , piece_id=2TZB3WZWPCWUXW36Z3JGCTFECKNC5OJPLBKLHB3U2VNEGYIYFKCA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.444232941 (589h4m53.315315986s) [noise=true , address=77.90.31.10:4518 , piece_id=2YFBS7RXT2LCS6T5JKXELD6XUZ3JJUJRGTIE3A6BKVUR6RUTGQJA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.447040297 (589h4m53.318123342s) [noise=true , address=145.79.227.3:4149 , piece_id=J4KTQR5BCQPFLZUVCPS3U6WNRJB47YGA5U5MVZJ5Y3RXTLTW5WPA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.449070974 (589h4m53.320154019s) [noise=true , address=77.90.31.12:4590 , piece_id=IOAXSQ42ROZT6V6I3V7L4E6SASHM7MH4GLSHR2BINBSBPV43QVJA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.450860581 (589h4m53.321943626s) [noise=true , address=77.90.31.11:4555 , piece_id=DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.452555569 (589h4m53.323638614s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.456353120 (589h4m53.327436165s) [noise=true , address=45.91.116.12:4226 , piece_id=O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.460079255 (589h4m53.3311623s) [noise=true , address=45.91.116.7:4009 , piece_id=YJBCELXP2YIZIBIYL3EV3XQ45FTVXRSLLJHPKHNQYMSOVDZ2MGOQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.464950641 (589h4m53.336033686s) [noise=true , address=145.79.227.2:4113 , piece_id=22AA2HEJ237XEOBQAHJ26N5J4FSNGP6HXMY5LM22FY44VRGKPHGQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:55.139659825 (589h4m55.010742871s) [upload=516409]: Next returned nil segment Dec 30 04:33:55.139680214 (589h4m55.010763259s) [upload=516409]: Waiting for error group managing segments... Dec 30 04:33:55.419515103 (589h4m55.290598148s) [noise=true , address=145.79.227.17:4972 , piece_id=6JB5VMQEIZMOGEULWAESCORSCPSAFVPMO4ZQ4A2KURERRUIDW7FA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.432174058 (589h4m55.303257093s) [noise=true , address=145.79.227.15:4873 , piece_id=H6V2GHW6WGYPSVZONKK63MP6T764OJXHMU53R3P4EMH6MWOEMNDA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.437302654 (589h4m55.308385689s) [noise=true , address=145.79.227.14:4837 , piece_id=T2N3GRL4RHBQW7RXS463IFLBKVOJASW66O3SKZOF4LORHPCUBHOQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.437565775 (589h4m55.30864882s) [noise=true , address=145.79.227.24:4582 , piece_id=2TZB3WZWPCWUXW36Z3JGCTFECKNC5OJPLBKLHB3U2VNEGYIYFKCA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.457133444 (589h4m55.328216489s) [noise=true , address=145.79.227.2:4113 , piece_id=22AA2HEJ237XEOBQAHJ26N5J4FSNGP6HXMY5LM22FY44VRGKPHGQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.457756110 (589h4m55.328839155s) [noise=true , address=145.79.227.3:4149 , piece_id=J4KTQR5BCQPFLZUVCPS3U6WNRJB47YGA5U5MVZJ5Y3RXTLTW5WPA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.604637414 (589h4m55.475720459s) [noise=true , address=145.79.227.35:4482 , piece_id=O4JKIDKYF2X3FDSCOQ5YFWZC6R4X3BKDPD6IOQ7GFBGLLVNIZYKQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.646242637 (589h4m55.517325672s) [noise=true , address=77.90.31.12:4590 , piece_id=IOAXSQ42ROZT6V6I3V7L4E6SASHM7MH4GLSHR2BINBSBPV43QVJA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.716537068 (589h4m55.587620113s) [noise=true , address=77.90.31.10:4518 , piece_id=2YFBS7RXT2LCS6T5JKXELD6XUZ3JJUJRGTIE3A6BKVUR6RUTGQJA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.843169556 (589h4m55.714252601s) [noise=true , address=45.91.116.7:4009 , piece_id=YJBCELXP2YIZIBIYL3EV3XQ45FTVXRSLLJHPKHNQYMSOVDZ2MGOQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.224883287 (589h4m56.095966332s) [noise=true , address=77.90.31.11:4555 , piece_id=DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.489291652 (589h4m56.360374707s) [noise=true , address=45.91.116.12:4226 , piece_id=O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.594859078 (589h4m56.465942124s) [noise=true , address=77.90.31.14:4661 , piece_id=Y2MZVSCLBQFWZ7RHRQ6ZPYHOZCJ5HY7MZ6KP4LYOKZBJRS5EZASQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.594881130 (589h4m56.465964175s) [seg_pos={0 0}, upload=516409]: Segment reached optimal threshold of 13 pieces. Dec 30 04:33:56.595107762 (589h4m56.466190797s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 04:33:56.595115196 (589h4m56.466198241s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595130866 (589h4m56.466213911s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 04:33:56.595144041 (589h4m56.466227117s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595385601 (589h4m56.466468636s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection Dec 30 04:33:56.595395931 (589h4m56.466478976s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595419927 (589h4m56.466502972s) [seg_pos={0 0}, upload=516409]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:33:56.595511090 (589h4m56.466594135s) [upload=516409]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:33:56.595517592 (589h4m56.466600637s) [upload=516409]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:33:56.891943372 (589h4m56.763026408s) [upload=516409]: Done uploading segments... Dec 30 04:33:56.891964744 (589h4m56.763047789s) [upload=516409]: Upload finished. err: <nil> Duration: 28.093830066s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..XXX............XX..X.....X..............X................X] duration: 5.000547174s trace: https://jaeger.dev.storj.io/trace/2a6809dd002dd615 FAIL (2025-12-30 04:35:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00057645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X............X..........................................] duration: 18.475582975s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:35:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 18.475605307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X..XXX............XX..X.....X..............X................] duration: 4.025007875s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:34:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.02503672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X............X.........................................] duration: 48.422921716s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:34:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 48.422937054s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XXX] duration: 30.00389035s trace: https://jaeger.dev.storj.io/trace/5d491286618aa306 FAIL (2025-12-30 04:34:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ) context deadline exceeded ======LOGS====== Dec 30 04:33:53.183104152 (589h4m53.054187197s) [upload=516409]: Starting upload Dec 30 04:33:53.183214783 (589h4m53.054297828s) [upload=516409]: Done starting upload Dec 30 04:33:53.183247104 (589h4m53.054330149s) [upload=516409]: Uploading segments... Dec 30 04:33:53.183636365 (589h4m53.0547194s) [upload=516409]: Got next segment. Inline: false Dec 30 04:33:53.183695148 (589h4m53.054778193s) [upload=516409]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:33:53.183718471 (589h4m53.054801506s) [upload=516409]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:33:53.430021290 (589h4m53.301104355s) [seg_pos={0 0}, upload=516409]: Begin upload segment... Dec 30 04:33:53.430115249 (589h4m53.301198294s) [seg_pos={0 0}, upload=516409]: Done begin upload segment. Dec 30 04:33:53.430177378 (589h4m53.301260413s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.430225990 (589h4m53.301309035s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.431398643 (589h4m53.302481688s) [noise=true , address=77.90.31.14:4661 , piece_id=Y2MZVSCLBQFWZ7RHRQ6ZPYHOZCJ5HY7MZ6KP4LYOKZBJRS5EZASQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.434681364 (589h4m53.305764459s) [noise=true , address=145.79.227.35:4482 , piece_id=O4JKIDKYF2X3FDSCOQ5YFWZC6R4X3BKDPD6IOQ7GFBGLLVNIZYKQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.435664345 (589h4m53.30674739s) [noise=true , address=145.79.227.14:4837 , piece_id=T2N3GRL4RHBQW7RXS463IFLBKVOJASW66O3SKZOF4LORHPCUBHOQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.436612571 (589h4m53.307695616s) [noise=true , address=145.79.227.15:4873 , piece_id=H6V2GHW6WGYPSVZONKK63MP6T764OJXHMU53R3P4EMH6MWOEMNDA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.437477207 (589h4m53.308560252s) [noise=true , address=145.79.227.17:4972 , piece_id=6JB5VMQEIZMOGEULWAESCORSCPSAFVPMO4ZQ4A2KURERRUIDW7FA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.440643055 (589h4m53.311726101s) [noise=true , address=145.79.227.24:4582 , piece_id=2TZB3WZWPCWUXW36Z3JGCTFECKNC5OJPLBKLHB3U2VNEGYIYFKCA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.444232941 (589h4m53.315315986s) [noise=true , address=77.90.31.10:4518 , piece_id=2YFBS7RXT2LCS6T5JKXELD6XUZ3JJUJRGTIE3A6BKVUR6RUTGQJA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.447040297 (589h4m53.318123342s) [noise=true , address=145.79.227.3:4149 , piece_id=J4KTQR5BCQPFLZUVCPS3U6WNRJB47YGA5U5MVZJ5Y3RXTLTW5WPA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.449070974 (589h4m53.320154019s) [noise=true , address=77.90.31.12:4590 , piece_id=IOAXSQ42ROZT6V6I3V7L4E6SASHM7MH4GLSHR2BINBSBPV43QVJA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.450860581 (589h4m53.321943626s) [noise=true , address=77.90.31.11:4555 , piece_id=DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.452555569 (589h4m53.323638614s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.456353120 (589h4m53.327436165s) [noise=true , address=45.91.116.12:4226 , piece_id=O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.460079255 (589h4m53.3311623s) [noise=true , address=45.91.116.7:4009 , piece_id=YJBCELXP2YIZIBIYL3EV3XQ45FTVXRSLLJHPKHNQYMSOVDZ2MGOQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:53.464950641 (589h4m53.336033686s) [noise=true , address=145.79.227.2:4113 , piece_id=22AA2HEJ237XEOBQAHJ26N5J4FSNGP6HXMY5LM22FY44VRGKPHGQ, seg_pos={0 0}, upload=516409]: Uploading piece... Dec 30 04:33:55.139659825 (589h4m55.010742871s) [upload=516409]: Next returned nil segment Dec 30 04:33:55.139680214 (589h4m55.010763259s) [upload=516409]: Waiting for error group managing segments... Dec 30 04:33:55.419515103 (589h4m55.290598148s) [noise=true , address=145.79.227.17:4972 , piece_id=6JB5VMQEIZMOGEULWAESCORSCPSAFVPMO4ZQ4A2KURERRUIDW7FA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.432174058 (589h4m55.303257093s) [noise=true , address=145.79.227.15:4873 , piece_id=H6V2GHW6WGYPSVZONKK63MP6T764OJXHMU53R3P4EMH6MWOEMNDA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.437302654 (589h4m55.308385689s) [noise=true , address=145.79.227.14:4837 , piece_id=T2N3GRL4RHBQW7RXS463IFLBKVOJASW66O3SKZOF4LORHPCUBHOQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.437565775 (589h4m55.30864882s) [noise=true , address=145.79.227.24:4582 , piece_id=2TZB3WZWPCWUXW36Z3JGCTFECKNC5OJPLBKLHB3U2VNEGYIYFKCA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.457133444 (589h4m55.328216489s) [noise=true , address=145.79.227.2:4113 , piece_id=22AA2HEJ237XEOBQAHJ26N5J4FSNGP6HXMY5LM22FY44VRGKPHGQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.457756110 (589h4m55.328839155s) [noise=true , address=145.79.227.3:4149 , piece_id=J4KTQR5BCQPFLZUVCPS3U6WNRJB47YGA5U5MVZJ5Y3RXTLTW5WPA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.604637414 (589h4m55.475720459s) [noise=true , address=145.79.227.35:4482 , piece_id=O4JKIDKYF2X3FDSCOQ5YFWZC6R4X3BKDPD6IOQ7GFBGLLVNIZYKQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.646242637 (589h4m55.517325672s) [noise=true , address=77.90.31.12:4590 , piece_id=IOAXSQ42ROZT6V6I3V7L4E6SASHM7MH4GLSHR2BINBSBPV43QVJA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.716537068 (589h4m55.587620113s) [noise=true , address=77.90.31.10:4518 , piece_id=2YFBS7RXT2LCS6T5JKXELD6XUZ3JJUJRGTIE3A6BKVUR6RUTGQJA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:55.843169556 (589h4m55.714252601s) [noise=true , address=45.91.116.7:4009 , piece_id=YJBCELXP2YIZIBIYL3EV3XQ45FTVXRSLLJHPKHNQYMSOVDZ2MGOQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.224883287 (589h4m56.095966332s) [noise=true , address=77.90.31.11:4555 , piece_id=DCWZWRL27JQNDPHSNU2TDZPDQO524ZEPHX7JUHQB2XKGX7NGQCTA, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.489291652 (589h4m56.360374707s) [noise=true , address=45.91.116.12:4226 , piece_id=O6Q34DGC4K5KZ2OUBQE7DMLMM6NSNKBMSNAEEBKMW6XFSVGG3MVQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.594859078 (589h4m56.465942124s) [noise=true , address=77.90.31.14:4661 , piece_id=Y2MZVSCLBQFWZ7RHRQ6ZPYHOZCJ5HY7MZ6KP4LYOKZBJRS5EZASQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: <nil> Dec 30 04:33:56.594881130 (589h4m56.465964175s) [seg_pos={0 0}, upload=516409]: Segment reached optimal threshold of 13 pieces. Dec 30 04:33:56.595107762 (589h4m56.466190797s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 04:33:56.595115196 (589h4m56.466198241s) [noise=true , address=45.91.116.8:4046 , piece_id=RUF4HYJW5J2VH2O2C7B4FPGIZNIB6ECTTRDZYMAU3L5XCG2JSN4A, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595130866 (589h4m56.466213911s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 04:33:56.595144041 (589h4m56.466227117s) [noise=true , address=145.79.227.44:4195 , piece_id=ET4WXMRJZ6XPVGD5WOTXDAPWB5WQTNQIESGM7VT42YUJPF2UOXKQ, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595385601 (589h4m56.466468636s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57376->77.90.31.13:4626: use of closed network connection Dec 30 04:33:56.595395931 (589h4m56.466478976s) [noise=true , address=77.90.31.13:4626 , piece_id=YILF2AQ7P22NMJFNUA3SO5PEEML2NKXIDORLAMLKBGLAXZKKGURQ, seg_pos={0 0}, upload=516409]: Long tail cancellation due to optimal threshold Dec 30 04:33:56.595419927 (589h4m56.466502972s) [seg_pos={0 0}, upload=516409]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:33:56.595511090 (589h4m56.466594135s) [upload=516409]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:33:56.595517592 (589h4m56.466600637s) [upload=516409]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:33:56.891943372 (589h4m56.763026408s) [upload=516409]: Done uploading segments... Dec 30 04:33:56.891964744 (589h4m56.763047789s) [upload=516409]: Upload finished. err: <nil> Duration: 30.003911963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X............X........................................] duration: 31.949669808s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:33:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 31.949687511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X..XXX............XX..X.....X..............X...............] duration: 3.53533909s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:33:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.535369367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X............X.......................................] duration: 48.489118562s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:33:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 48.489151636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...XX] duration: 30.001857787s trace: https://jaeger.dev.storj.io/trace/2d051fd5f45c81b2 FAIL (2025-12-30 04:33:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:32:48.726642461 (589h3m48.597725506s) [upload=516392]: Starting upload Dec 30 04:32:48.727126703 (589h3m48.598209758s) [upload=516392]: Done starting upload Dec 30 04:32:48.727212547 (589h3m48.598295592s) [upload=516392]: Uploading segments... Dec 30 04:32:48.727584114 (589h3m48.598667159s) [upload=516392]: Got next segment. Inline: false Dec 30 04:32:48.727708021 (589h3m48.598791066s) [upload=516392]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:32:48.727716166 (589h3m48.598799201s) [upload=516392]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:32:48.971499187 (589h3m48.842582222s) [seg_pos={0 0}, upload=516392]: Begin upload segment... Dec 30 04:32:48.971560614 (589h3m48.842643649s) [seg_pos={0 0}, upload=516392]: Done begin upload segment. Dec 30 04:32:48.971605699 (589h3m48.842688735s) [noise=true , address=77.90.31.17:4773 , piece_id=W7CF7HXQVS7RNKYMXWKNQN3DU4INGFWU7C7CJO2TL7GWGBK4ZVBQ, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.971699408 (589h3m48.842782453s) [noise=true , address=145.79.227.10:4402 , piece_id=XVKD32DMMRWLEJMM7VAGAMX6KTWAWAKPD2ARSMJLLU3CPW3BTLGA, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.972190854 (589h3m48.843273889s) [noise=true , address=45.91.116.5:4297 , piece_id=BV4SEIPWOBYV4FC2GJUOB7FU3DJ3OS3H7ACJLMTWH7GBCN3KFAEQ, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.972595104 (589h3m48.843678149s) [noise=true , address=45.91.116.6:4974 , piece_id=B6CBOVVGEUXSGRUWIFDGMD4C5W2CP2UQRDJTLIAQEVCPA2GD7L6Q, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.973177803 (589h3m48.844260848s) [noise=true , address=145.79.227.19:4078 , piece_id=L6U4ISRQCQI6FRD6REPXID6Q5SSG6JCXAVRMHWOP2H4AM3X2GYJA, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.973593765 (589h3m48.8446768s) [noise=true , address=145.79.227.4:4186 , piece_id=VAGYDHNUDLNUDED2ARRBTQQ6M734ESOR2YT7OYFJLAW3RUJCY4MA, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.974038181 (589h3m48.845121227s) [noise=true , address=145.79.227.45:4229 , piece_id=VXPM26KHSUNH5M6UJLJUYLJZBDNND2W6ASOHE6CNG4ZCOISRIB7A, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.974543905 (589h3m48.84562695s) [noise=true , address=145.79.227.11:4731 , piece_id=UNXD3HJBD2NYB5S2MBGELBU76BHWBFKH5VVSN6CWSPYDI7X763LQ, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.975950864 (589h3m48.847033909s) [noise=true , address=145.79.227.46:4602 , piece_id=CJQKRJKSS3Q7OHY42BJNLXFBMCC3WMTAA73CSCSK2ZF2ZXQSBM6Q, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.975987433 (589h3m48.847070468s) [noise=true , address=145.79.227.47:4266 , piece_id=FCBHHXCJFOVLQQ7VF2REGR55RFY66NO33YSMLLCHIYAVJLPR2TVQ, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.977506847 (589h3m48.848589892s) [noise=true , address=145.79.227.34:4907 , piece_id=RLOZF2IHFU4647P4UYQMFRZTGJFPP57LFJCMVTRYSFMN4ZGGH6YA, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.979917867 (589h3m48.851000913s) [noise=true , address=77.90.31.12:4589 , piece_id=MOMJYBOTEP2BY42ZMEEUYDKM5Z2YZASTQRHBIRWQGWA6CL7FX5EA, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.981362027 (589h3m48.852445072s) [noise=true , address=45.91.116.9:4264 , piece_id=POR4DWSJJLYADT4JVTXKKQ47MMJETXATMFDTYJEA3GKDLY3APY7A, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.982724552 (589h3m48.853807597s) [noise=true , address=145.79.227.32:4834 , piece_id=DKBLCGDBGS7FWLXEA5WMJ6AZM4SDQNPW6HTOR7Y4TH7Z2TXKRKBQ, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.984061868 (589h3m48.855144903s) [noise=true , address=145.79.227.39:4014 , piece_id=E2ASRIUFMEYPDC7JBQK2OU4VCPDDEAHG7K7THYW7QO7YUID4XOZQ, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:48.984536613 (589h3m48.855619648s) [noise=true , address=77.90.31.21:4880 , piece_id=ODWBFRZGHOI6AH4D6ZY7L24TGE65IRLGUGM7VOQHWV7NE7A5KV5Q, seg_pos={0 0}, upload=516392]: Uploading piece... Dec 30 04:32:50.681598681 (589h3m50.552681716s) [upload=516392]: Next returned nil segment Dec 30 04:32:50.681620052 (589h3m50.552703087s) [upload=516392]: Waiting for error group managing segments... Dec 30 04:32:50.966021405 (589h3m50.83710444s) [noise=true , address=145.79.227.19:4078 , piece_id=L6U4ISRQCQI6FRD6REPXID6Q5SSG6JCXAVRMHWOP2H4AM3X2GYJA, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:50.970279333 (589h3m50.841362378s) [noise=true , address=145.79.227.4:4186 , piece_id=VAGYDHNUDLNUDED2ARRBTQQ6M734ESOR2YT7OYFJLAW3RUJCY4MA, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:50.970951483 (589h3m50.842034528s) [noise=true , address=145.79.227.45:4229 , piece_id=VXPM26KHSUNH5M6UJLJUYLJZBDNND2W6ASOHE6CNG4ZCOISRIB7A, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:50.972174903 (589h3m50.843257938s) [noise=true , address=145.79.227.46:4602 , piece_id=CJQKRJKSS3Q7OHY42BJNLXFBMCC3WMTAA73CSCSK2ZF2ZXQSBM6Q, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:50.988080118 (589h3m50.859163163s) [noise=true , address=145.79.227.11:4731 , piece_id=UNXD3HJBD2NYB5S2MBGELBU76BHWBFKH5VVSN6CWSPYDI7X763LQ, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:50.989070565 (589h3m50.86015361s) [noise=true , address=145.79.227.47:4266 , piece_id=FCBHHXCJFOVLQQ7VF2REGR55RFY66NO33YSMLLCHIYAVJLPR2TVQ, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:50.993946058 (589h3m50.865029103s) [noise=true , address=145.79.227.10:4402 , piece_id=XVKD32DMMRWLEJMM7VAGAMX6KTWAWAKPD2ARSMJLLU3CPW3BTLGA, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:50.999576720 (589h3m50.870659755s) [noise=true , address=145.79.227.39:4014 , piece_id=E2ASRIUFMEYPDC7JBQK2OU4VCPDDEAHG7K7THYW7QO7YUID4XOZQ, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:51.013087035 (589h3m50.88417007s) [noise=true , address=145.79.227.32:4834 , piece_id=DKBLCGDBGS7FWLXEA5WMJ6AZM4SDQNPW6HTOR7Y4TH7Z2TXKRKBQ, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:51.218175385 (589h3m51.08925843s) [noise=true , address=45.91.116.9:4264 , piece_id=POR4DWSJJLYADT4JVTXKKQ47MMJETXATMFDTYJEA3GKDLY3APY7A, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:51.273011812 (589h3m51.144094857s) [noise=true , address=77.90.31.17:4773 , piece_id=W7CF7HXQVS7RNKYMXWKNQN3DU4INGFWU7C7CJO2TL7GWGBK4ZVBQ, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:51.285177879 (589h3m51.156260914s) [noise=true , address=45.91.116.5:4297 , piece_id=BV4SEIPWOBYV4FC2GJUOB7FU3DJ3OS3H7ACJLMTWH7GBCN3KFAEQ, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:51.322546233 (589h3m51.193629278s) [noise=true , address=77.90.31.21:4880 , piece_id=ODWBFRZGHOI6AH4D6ZY7L24TGE65IRLGUGM7VOQHWV7NE7A5KV5Q, seg_pos={0 0}, upload=516392]: Done uploading piece. err: <nil> Dec 30 04:32:51.322567564 (589h3m51.193650599s) [seg_pos={0 0}, upload=516392]: Segment reached optimal threshold of 13 pieces. Dec 30 04:32:51.322772475 (589h3m51.19385552s) [noise=true , address=77.90.31.12:4589 , piece_id=MOMJYBOTEP2BY42ZMEEUYDKM5Z2YZASTQRHBIRWQGWA6CL7FX5EA, seg_pos={0 0}, upload=516392]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 30 04:32:51.322779678 (589h3m51.193862713s) [noise=true , address=77.90.31.12:4589 , piece_id=MOMJYBOTEP2BY42ZMEEUYDKM5Z2YZASTQRHBIRWQGWA6CL7FX5EA, seg_pos={0 0}, upload=516392]: Long tail cancellation due to optimal threshold Dec 30 04:32:51.323090300 (589h3m51.194173345s) [noise=true , address=45.91.116.6:4974 , piece_id=B6CBOVVGEUXSGRUWIFDGMD4C5W2CP2UQRDJTLIAQEVCPA2GD7L6Q, seg_pos={0 0}, upload=516392]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:33252->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:33252->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33252->45.91.116.6:4974: use of closed network connection Dec 30 04:32:51.323097814 (589h3m51.194180859s) [noise=true , address=45.91.116.6:4974 , piece_id=B6CBOVVGEUXSGRUWIFDGMD4C5W2CP2UQRDJTLIAQEVCPA2GD7L6Q, seg_pos={0 0}, upload=516392]: Long tail cancellation due to optimal threshold Dec 30 04:32:51.323109195 (589h3m51.19419225s) [noise=true , address=145.79.227.34:4907 , piece_id=RLOZF2IHFU4647P4UYQMFRZTGJFPP57LFJCMVTRYSFMN4ZGGH6YA, seg_pos={0 0}, upload=516392]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 04:32:51.323140545 (589h3m51.19422359s) [noise=true , address=145.79.227.34:4907 , piece_id=RLOZF2IHFU4647P4UYQMFRZTGJFPP57LFJCMVTRYSFMN4ZGGH6YA, seg_pos={0 0}, upload=516392]: Long tail cancellation due to optimal threshold Dec 30 04:32:51.323206300 (589h3m51.194289335s) [seg_pos={0 0}, upload=516392]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:32:51.323281784 (589h3m51.194364829s) [upload=516392]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:32:51.323287035 (589h3m51.19437008s) [upload=516392]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:32:51.617131270 (589h3m51.488214315s) [upload=516392]: Done uploading segments... Dec 30 04:32:51.617150436 (589h3m51.488233481s) [upload=516392]: Upload finished. err: <nil> Duration: 30.00188027s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X..XXX............XX..X.....X..............X..............] duration: 4.459715336s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:32:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.459740915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X............X......................................] duration: 50.628720649s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:32:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 50.628742199s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X..XXX............XX..X.....X..............X.............] duration: 4.088728527s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:32:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.088765919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...X] duration: 30.002757425s trace: https://jaeger.dev.storj.io/trace/466c43f3091a030b FAIL (2025-12-30 04:32:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:31:49.223288325 (589h2m49.094371371s) [upload=516381]: Starting upload Dec 30 04:31:49.223366314 (589h2m49.094449349s) [upload=516381]: Done starting upload Dec 30 04:31:49.223405358 (589h2m49.094488403s) [upload=516381]: Uploading segments... Dec 30 04:31:49.223603606 (589h2m49.094686651s) [upload=516381]: Got next segment. Inline: false Dec 30 04:31:49.223643642 (589h2m49.094726687s) [upload=516381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:31:49.223648551 (589h2m49.094731596s) [upload=516381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:31:49.468686441 (589h2m49.339769496s) [seg_pos={0 0}, upload=516381]: Begin upload segment... Dec 30 04:31:49.468756966 (589h2m49.339840011s) [seg_pos={0 0}, upload=516381]: Done begin upload segment. Dec 30 04:31:49.468829844 (589h2m49.339912889s) [noise=true , address=77.90.31.19:4842 , piece_id=GWJFPMFAGS6N3ZVHX5ZWY5RQDJEEN5XCLUMZRM4KG25EVT2SDWDA, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.468846626 (589h2m49.339929671s) [noise=true , address=145.79.227.38:4977 , piece_id=P3IYY3PV7XZ6CRUENRUDVNQLAKOFCRKTJ4UHRGM76SFKJLJTDY3A, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.469717644 (589h2m49.340800689s) [noise=true , address=45.91.116.10:4082 , piece_id=P3QP46XI22Y4KSEO4MBY57R73FHBSQZHIJ3TMNQJJR6IVUNP7X2Q, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.470197929 (589h2m49.341280974s) [noise=true , address=145.79.227.47:4265 , piece_id=O4SRP5M75R3TDMEB5O7Y56TL3BCPPJGA3HZHAT247PC7T5IZ4NNA, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.470732788 (589h2m49.341815823s) [noise=true , address=145.79.227.19:4078 , piece_id=C3YSV7X5544C3623SREIKTMEIV647RFW7HZOZ5ZW5ZGLMVDKHJJA, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.471175551 (589h2m49.342258596s) [noise=true , address=145.79.227.13:4802 , piece_id=OCC7WFVO37PABU3QVMP2GQY4AQ3WZRXH7QKIUBQEKSHXO5VDXLKQ, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.471603626 (589h2m49.342686671s) [noise=true , address=145.79.227.14:4838 , piece_id=WNPZ4CUOM3C5HRND67UXX4LG3ZCYF5JVMFPEPWTPBSDUH7ELPNZQ, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.472006583 (589h2m49.343089628s) [noise=true , address=145.79.227.50:4375 , piece_id=IVGIVY6ZFHSQBEXFUBNZSFD3KFMNRYZEVZ2IIL5LIYBVFPMVFIKA, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.472412376 (589h2m49.343495411s) [noise=true , address=45.91.116.5:4298 , piece_id=IN3M4QWGEE7UKDDT22WCJZL36OWU2WNNBHV2J2YKTPHXL7OZFCDQ, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.472818589 (589h2m49.343901634s) [noise=true , address=77.90.31.10:4518 , piece_id=M2HFXJYRW5UXWWL4AQC5RMSI5QTCSLGJQNJWEZAI5OPSUBLNAURQ, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.473562596 (589h2m49.344645631s) [noise=true , address=145.79.227.37:4566 , piece_id=RBVTRFOFYSHGPY45XKE7N4M5LVXGI5AOZBSC3EXLN2TKNXER563A, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.474110800 (589h2m49.345193845s) [noise=true , address=145.79.227.26:4655 , piece_id=AHXEQZLAKMB36VWCPCXVS27FER35FCYOSLKHLKWTUO4RC2E6GXPQ, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.474580043 (589h2m49.345663089s) [noise=true , address=77.90.31.21:4878 , piece_id=OLGXYVBHXI4N6RFOGY2BT3PBZSG5OY7QXZ66WUYQRUHY5NUEKPSA, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.475004431 (589h2m49.346087476s) [noise=true , address=145.79.227.2:4115 , piece_id=MKJCFAS4EUWI4GLZOR6VLUFWSUQMEGXKPBBOV5P7U7T2PV2DPOBA, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.475854451 (589h2m49.346937486s) [noise=true , address=145.79.227.48:4301 , piece_id=LZPI6ICYY7TWZJAMINMDJEQDPHMPEIXCPQYX6BVOKNX5EWPVJLUQ, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:49.476911232 (589h2m49.347994268s) [noise=true , address=145.79.227.45:4229 , piece_id=4BM7E3MXDHM5JHGV72JW332LW7TMF3GDEQAAXAAWS6ZE4JL44K6A, seg_pos={0 0}, upload=516381]: Uploading piece... Dec 30 04:31:51.181114868 (589h2m51.052197903s) [upload=516381]: Next returned nil segment Dec 30 04:31:51.181136640 (589h2m51.052219675s) [upload=516381]: Waiting for error group managing segments... Dec 30 04:31:51.463443767 (589h2m51.334526802s) [noise=true , address=145.79.227.19:4078 , piece_id=C3YSV7X5544C3623SREIKTMEIV647RFW7HZOZ5ZW5ZGLMVDKHJJA, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.464924325 (589h2m51.336007371s) [noise=true , address=145.79.227.26:4655 , piece_id=AHXEQZLAKMB36VWCPCXVS27FER35FCYOSLKHLKWTUO4RC2E6GXPQ, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.466170198 (589h2m51.337253243s) [noise=true , address=145.79.227.14:4838 , piece_id=WNPZ4CUOM3C5HRND67UXX4LG3ZCYF5JVMFPEPWTPBSDUH7ELPNZQ, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.471270519 (589h2m51.342353554s) [noise=true , address=145.79.227.48:4301 , piece_id=LZPI6ICYY7TWZJAMINMDJEQDPHMPEIXCPQYX6BVOKNX5EWPVJLUQ, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.481373257 (589h2m51.352456302s) [noise=true , address=145.79.227.38:4977 , piece_id=P3IYY3PV7XZ6CRUENRUDVNQLAKOFCRKTJ4UHRGM76SFKJLJTDY3A, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.482860689 (589h2m51.353943734s) [noise=true , address=145.79.227.13:4802 , piece_id=OCC7WFVO37PABU3QVMP2GQY4AQ3WZRXH7QKIUBQEKSHXO5VDXLKQ, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.488352276 (589h2m51.359435321s) [noise=true , address=145.79.227.37:4566 , piece_id=RBVTRFOFYSHGPY45XKE7N4M5LVXGI5AOZBSC3EXLN2TKNXER563A, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.491173098 (589h2m51.362256143s) [noise=true , address=145.79.227.45:4229 , piece_id=4BM7E3MXDHM5JHGV72JW332LW7TMF3GDEQAAXAAWS6ZE4JL44K6A, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.493788276 (589h2m51.364871311s) [noise=true , address=145.79.227.50:4375 , piece_id=IVGIVY6ZFHSQBEXFUBNZSFD3KFMNRYZEVZ2IIL5LIYBVFPMVFIKA, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.501296263 (589h2m51.372379308s) [noise=true , address=145.79.227.47:4265 , piece_id=O4SRP5M75R3TDMEB5O7Y56TL3BCPPJGA3HZHAT247PC7T5IZ4NNA, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.542413837 (589h2m51.413496882s) [noise=true , address=145.79.227.2:4115 , piece_id=MKJCFAS4EUWI4GLZOR6VLUFWSUQMEGXKPBBOV5P7U7T2PV2DPOBA, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.631610511 (589h2m51.502693556s) [noise=true , address=77.90.31.19:4842 , piece_id=GWJFPMFAGS6N3ZVHX5ZWY5RQDJEEN5XCLUMZRM4KG25EVT2SDWDA, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.690117568 (589h2m51.561200613s) [noise=true , address=45.91.116.5:4298 , piece_id=IN3M4QWGEE7UKDDT22WCJZL36OWU2WNNBHV2J2YKTPHXL7OZFCDQ, seg_pos={0 0}, upload=516381]: Done uploading piece. err: <nil> Dec 30 04:31:51.690139731 (589h2m51.561222776s) [seg_pos={0 0}, upload=516381]: Segment reached optimal threshold of 13 pieces. Dec 30 04:31:51.690456594 (589h2m51.561539629s) [noise=true , address=77.90.31.10:4518 , piece_id=M2HFXJYRW5UXWWL4AQC5RMSI5QTCSLGJQNJWEZAI5OPSUBLNAURQ, seg_pos={0 0}, upload=516381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 04:31:51.690463547 (589h2m51.561546582s) [noise=true , address=77.90.31.10:4518 , piece_id=M2HFXJYRW5UXWWL4AQC5RMSI5QTCSLGJQNJWEZAI5OPSUBLNAURQ, seg_pos={0 0}, upload=516381]: Long tail cancellation due to optimal threshold Dec 30 04:31:51.690589608 (589h2m51.561672643s) [noise=true , address=77.90.31.21:4878 , piece_id=OLGXYVBHXI4N6RFOGY2BT3PBZSG5OY7QXZ66WUYQRUHY5NUEKPSA, seg_pos={0 0}, upload=516381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 30 04:31:51.690597011 (589h2m51.561680046s) [noise=true , address=77.90.31.21:4878 , piece_id=OLGXYVBHXI4N6RFOGY2BT3PBZSG5OY7QXZ66WUYQRUHY5NUEKPSA, seg_pos={0 0}, upload=516381]: Long tail cancellation due to optimal threshold Dec 30 04:31:51.690626688 (589h2m51.561709723s) [noise=true , address=45.91.116.10:4082 , piece_id=P3QP46XI22Y4KSEO4MBY57R73FHBSQZHIJ3TMNQJJR6IVUNP7X2Q, seg_pos={0 0}, upload=516381]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 30 04:31:51.690633681 (589h2m51.561716726s) [noise=true , address=45.91.116.10:4082 , piece_id=P3QP46XI22Y4KSEO4MBY57R73FHBSQZHIJ3TMNQJJR6IVUNP7X2Q, seg_pos={0 0}, upload=516381]: Long tail cancellation due to optimal threshold Dec 30 04:31:51.690648549 (589h2m51.561731584s) [seg_pos={0 0}, upload=516381]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:31:51.690717280 (589h2m51.561800315s) [upload=516381]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:31:51.690722440 (589h2m51.561805485s) [upload=516381]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:31:51.982680673 (589h2m51.853763718s) [upload=516381]: Done uploading segments... Dec 30 04:31:51.982699769 (589h2m51.853782804s) [upload=516381]: Upload finished. err: <nil> Duration: 30.002782563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X............X.....................................] duration: 45.241934151s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 45.241962844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X..XXX............XX..X.....X..............X............] duration: 4.416058034s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:31:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.416136123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX...] duration: 5.954392006s trace: https://jaeger.dev.storj.io/trace/4a5d023bb2664f91 FAIL (2025-12-30 04:31:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:29:28.434399217 (589h0m28.305482262s) [upload=516347]: Starting upload Dec 30 04:29:28.434489869 (589h0m28.305572914s) [upload=516347]: Done starting upload Dec 30 04:29:28.434617622 (589h0m28.305700657s) [upload=516347]: Uploading segments... Dec 30 04:29:28.434789019 (589h0m28.305872054s) [upload=516347]: Got next segment. Inline: false Dec 30 04:29:28.434836589 (589h0m28.305919624s) [upload=516347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:29:28.434845076 (589h0m28.305928121s) [upload=516347]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:29:28.682545798 (589h0m28.553628833s) [seg_pos={0 0}, upload=516347]: Begin upload segment... Dec 30 04:29:28.682596335 (589h0m28.55367938s) [seg_pos={0 0}, upload=516347]: Done begin upload segment. Dec 30 04:29:28.682619108 (589h0m28.553702153s) [noise=true , address=145.79.227.39:4015 , piece_id=I6ZIVSJERWVISYX626AOLJKQHNBTCRSTDNPQL3FBDZCKR5SJHSWQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.682630059 (589h0m28.553713094s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683215334 (589h0m28.554298369s) [noise=true , address=145.79.227.51:4411 , piece_id=F76JVREPOILNESWKXX7HDLP4Y2BSTD5D3HC7T7ZVGR6JXNRILH4A, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683544099 (589h0m28.554627134s) [noise=true , address=77.90.31.16:4735 , piece_id=OBPAHXCEQ6JCQ2TTSJS6HVOKZIV3MWXM3I2ITOTNKFALIRJBY2NA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683958839 (589h0m28.555041884s) [noise=true , address=145.79.227.52:4446 , piece_id=R6XOOOZBHBSLDZXYUVJ4EHNBDDY26YQOJ6FKJZQITOJCJOZ57VBQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.684284729 (589h0m28.555367774s) [noise=true , address=145.79.227.14:4837 , piece_id=GNQTXNCXEIYBHWLZCGMVV5HX67RVPKJQ37YZZYGZSQ5G4BO67ZVQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.684695121 (589h0m28.555778166s) [noise=true , address=77.90.31.17:4773 , piece_id=DQJCHBNPDKNCTFW2JNZWFBP5UCJRF57YE6QP4LSTHMBI7N5PA7XA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685041461 (589h0m28.556124496s) [noise=true , address=45.91.116.8:4048 , piece_id=UOP4NIRWNOPAHXLBIJ65HSIMRHFHE7RV2O7AY5BX6GDND3WM7BYA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685431825 (589h0m28.55651486s) [noise=true , address=145.79.227.8:4331 , piece_id=GFQBP7V5M4X67AASKKPBZ6QNGNULV62LVJWHQB7QVWXEPYFEQDQA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685760029 (589h0m28.556843074s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.686501010 (589h0m28.557584055s) [noise=true , address=145.79.227.11:4731 , piece_id=L3W4RC3BI6TTIP5VXKCY4IU2DNPHQUQ5A4YE4POHUNCDO2LSHS5Q, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.687010951 (589h0m28.558093996s) [noise=true , address=145.79.227.27:4689 , piece_id=65444YOGN66UYSRBELQCWW6I7V5BMT3TWSZX4HY5QK2G6GOR2K2A, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.687508569 (589h0m28.558591604s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688010144 (589h0m28.559093179s) [noise=true , address=145.79.227.16:4912 , piece_id=BDX7LJ5YUDKEKV467DWTQUEL7DHUGM3D7AX2XXPLTN2OXRIYNCHA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688470541 (589h0m28.559553576s) [noise=true , address=77.90.31.19:4844 , piece_id=POZADDPXUYAHN4XSDUXVD6BSFHDDC6QPGSPQFV2TGT4775FT6DAA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688966615 (589h0m28.56004966s) [noise=true , address=145.79.227.46:4601 , piece_id=56ED5TCCWZ5WHBCJQDTDHOC2WUPM6XJU2P24WDBG5BB2BTCZ5Q7Q, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:30.295075614 (589h0m30.166158649s) [upload=516347]: Next returned nil segment Dec 30 04:29:30.295090643 (589h0m30.166173698s) [upload=516347]: Waiting for error group managing segments... Dec 30 04:29:30.676109141 (589h0m30.547192176s) [noise=true , address=145.79.227.11:4731 , piece_id=L3W4RC3BI6TTIP5VXKCY4IU2DNPHQUQ5A4YE4POHUNCDO2LSHS5Q, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.680628126 (589h0m30.551711171s) [noise=true , address=145.79.227.46:4601 , piece_id=56ED5TCCWZ5WHBCJQDTDHOC2WUPM6XJU2P24WDBG5BB2BTCZ5Q7Q, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.685669807 (589h0m30.556752852s) [noise=true , address=145.79.227.14:4837 , piece_id=GNQTXNCXEIYBHWLZCGMVV5HX67RVPKJQ37YZZYGZSQ5G4BO67ZVQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.686564291 (589h0m30.557647336s) [noise=true , address=145.79.227.16:4912 , piece_id=BDX7LJ5YUDKEKV467DWTQUEL7DHUGM3D7AX2XXPLTN2OXRIYNCHA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.690018117 (589h0m30.561101162s) [noise=true , address=145.79.227.27:4689 , piece_id=65444YOGN66UYSRBELQCWW6I7V5BMT3TWSZX4HY5QK2G6GOR2K2A, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.699105572 (589h0m30.570188607s) [noise=true , address=145.79.227.51:4411 , piece_id=F76JVREPOILNESWKXX7HDLP4Y2BSTD5D3HC7T7ZVGR6JXNRILH4A, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.700519285 (589h0m30.57160232s) [noise=true , address=145.79.227.52:4446 , piece_id=R6XOOOZBHBSLDZXYUVJ4EHNBDDY26YQOJ6FKJZQITOJCJOZ57VBQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.703589371 (589h0m30.574672406s) [noise=true , address=145.79.227.8:4331 , piece_id=GFQBP7V5M4X67AASKKPBZ6QNGNULV62LVJWHQB7QVWXEPYFEQDQA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.703806304 (589h0m30.574889339s) [noise=true , address=145.79.227.39:4015 , piece_id=I6ZIVSJERWVISYX626AOLJKQHNBTCRSTDNPQL3FBDZCKR5SJHSWQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.880526099 (589h0m30.751609144s) [noise=true , address=77.90.31.16:4735 , piece_id=OBPAHXCEQ6JCQ2TTSJS6HVOKZIV3MWXM3I2ITOTNKFALIRJBY2NA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.965541162 (589h0m30.836624197s) [noise=true , address=77.90.31.17:4773 , piece_id=DQJCHBNPDKNCTFW2JNZWFBP5UCJRF57YE6QP4LSTHMBI7N5PA7XA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.990980616 (589h0m30.862063651s) [noise=true , address=77.90.31.19:4844 , piece_id=POZADDPXUYAHN4XSDUXVD6BSFHDDC6QPGSPQFV2TGT4775FT6DAA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:31.015210327 (589h0m30.886293372s) [noise=true , address=45.91.116.8:4048 , piece_id=UOP4NIRWNOPAHXLBIJ65HSIMRHFHE7RV2O7AY5BX6GDND3WM7BYA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:31.015233510 (589h0m30.886316555s) [seg_pos={0 0}, upload=516347]: Segment reached optimal threshold of 13 pieces. Dec 30 04:29:31.015437749 (589h0m30.886520784s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection Dec 30 04:29:31.015444632 (589h0m30.886527667s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015598556 (589h0m30.886681591s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:31.015604958 (589h0m30.886688003s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015679870 (589h0m30.886762915s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:31.015702764 (589h0m30.886785829s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015745716 (589h0m30.886828761s) [seg_pos={0 0}, upload=516347]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:29:31.015896483 (589h0m30.886979528s) [upload=516347]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:29:31.015906552 (589h0m30.886989587s) [upload=516347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:29:31.343639619 (589h0m31.214722664s) [upload=516347]: Done uploading segments... Dec 30 04:29:31.343662031 (589h0m31.214745076s) [upload=516347]: Upload finished. err: <nil> Duration: 5.954526774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X............X....................................] duration: 15.382566287s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:30:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 15.382616723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....X..XXX............XX..X.....X..............X...........] duration: 5.000608482s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:30:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000630655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX..] duration: 6.007938886s trace: https://jaeger.dev.storj.io/trace/4a5d023bb2664f91 FAIL (2025-12-30 04:30:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:29:28.434399217 (589h0m28.305482262s) [upload=516347]: Starting upload Dec 30 04:29:28.434489869 (589h0m28.305572914s) [upload=516347]: Done starting upload Dec 30 04:29:28.434617622 (589h0m28.305700657s) [upload=516347]: Uploading segments... Dec 30 04:29:28.434789019 (589h0m28.305872054s) [upload=516347]: Got next segment. Inline: false Dec 30 04:29:28.434836589 (589h0m28.305919624s) [upload=516347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:29:28.434845076 (589h0m28.305928121s) [upload=516347]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:29:28.682545798 (589h0m28.553628833s) [seg_pos={0 0}, upload=516347]: Begin upload segment... Dec 30 04:29:28.682596335 (589h0m28.55367938s) [seg_pos={0 0}, upload=516347]: Done begin upload segment. Dec 30 04:29:28.682619108 (589h0m28.553702153s) [noise=true , address=145.79.227.39:4015 , piece_id=I6ZIVSJERWVISYX626AOLJKQHNBTCRSTDNPQL3FBDZCKR5SJHSWQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.682630059 (589h0m28.553713094s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683215334 (589h0m28.554298369s) [noise=true , address=145.79.227.51:4411 , piece_id=F76JVREPOILNESWKXX7HDLP4Y2BSTD5D3HC7T7ZVGR6JXNRILH4A, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683544099 (589h0m28.554627134s) [noise=true , address=77.90.31.16:4735 , piece_id=OBPAHXCEQ6JCQ2TTSJS6HVOKZIV3MWXM3I2ITOTNKFALIRJBY2NA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683958839 (589h0m28.555041884s) [noise=true , address=145.79.227.52:4446 , piece_id=R6XOOOZBHBSLDZXYUVJ4EHNBDDY26YQOJ6FKJZQITOJCJOZ57VBQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.684284729 (589h0m28.555367774s) [noise=true , address=145.79.227.14:4837 , piece_id=GNQTXNCXEIYBHWLZCGMVV5HX67RVPKJQ37YZZYGZSQ5G4BO67ZVQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.684695121 (589h0m28.555778166s) [noise=true , address=77.90.31.17:4773 , piece_id=DQJCHBNPDKNCTFW2JNZWFBP5UCJRF57YE6QP4LSTHMBI7N5PA7XA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685041461 (589h0m28.556124496s) [noise=true , address=45.91.116.8:4048 , piece_id=UOP4NIRWNOPAHXLBIJ65HSIMRHFHE7RV2O7AY5BX6GDND3WM7BYA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685431825 (589h0m28.55651486s) [noise=true , address=145.79.227.8:4331 , piece_id=GFQBP7V5M4X67AASKKPBZ6QNGNULV62LVJWHQB7QVWXEPYFEQDQA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685760029 (589h0m28.556843074s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.686501010 (589h0m28.557584055s) [noise=true , address=145.79.227.11:4731 , piece_id=L3W4RC3BI6TTIP5VXKCY4IU2DNPHQUQ5A4YE4POHUNCDO2LSHS5Q, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.687010951 (589h0m28.558093996s) [noise=true , address=145.79.227.27:4689 , piece_id=65444YOGN66UYSRBELQCWW6I7V5BMT3TWSZX4HY5QK2G6GOR2K2A, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.687508569 (589h0m28.558591604s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688010144 (589h0m28.559093179s) [noise=true , address=145.79.227.16:4912 , piece_id=BDX7LJ5YUDKEKV467DWTQUEL7DHUGM3D7AX2XXPLTN2OXRIYNCHA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688470541 (589h0m28.559553576s) [noise=true , address=77.90.31.19:4844 , piece_id=POZADDPXUYAHN4XSDUXVD6BSFHDDC6QPGSPQFV2TGT4775FT6DAA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688966615 (589h0m28.56004966s) [noise=true , address=145.79.227.46:4601 , piece_id=56ED5TCCWZ5WHBCJQDTDHOC2WUPM6XJU2P24WDBG5BB2BTCZ5Q7Q, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:30.295075614 (589h0m30.166158649s) [upload=516347]: Next returned nil segment Dec 30 04:29:30.295090643 (589h0m30.166173698s) [upload=516347]: Waiting for error group managing segments... Dec 30 04:29:30.676109141 (589h0m30.547192176s) [noise=true , address=145.79.227.11:4731 , piece_id=L3W4RC3BI6TTIP5VXKCY4IU2DNPHQUQ5A4YE4POHUNCDO2LSHS5Q, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.680628126 (589h0m30.551711171s) [noise=true , address=145.79.227.46:4601 , piece_id=56ED5TCCWZ5WHBCJQDTDHOC2WUPM6XJU2P24WDBG5BB2BTCZ5Q7Q, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.685669807 (589h0m30.556752852s) [noise=true , address=145.79.227.14:4837 , piece_id=GNQTXNCXEIYBHWLZCGMVV5HX67RVPKJQ37YZZYGZSQ5G4BO67ZVQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.686564291 (589h0m30.557647336s) [noise=true , address=145.79.227.16:4912 , piece_id=BDX7LJ5YUDKEKV467DWTQUEL7DHUGM3D7AX2XXPLTN2OXRIYNCHA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.690018117 (589h0m30.561101162s) [noise=true , address=145.79.227.27:4689 , piece_id=65444YOGN66UYSRBELQCWW6I7V5BMT3TWSZX4HY5QK2G6GOR2K2A, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.699105572 (589h0m30.570188607s) [noise=true , address=145.79.227.51:4411 , piece_id=F76JVREPOILNESWKXX7HDLP4Y2BSTD5D3HC7T7ZVGR6JXNRILH4A, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.700519285 (589h0m30.57160232s) [noise=true , address=145.79.227.52:4446 , piece_id=R6XOOOZBHBSLDZXYUVJ4EHNBDDY26YQOJ6FKJZQITOJCJOZ57VBQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.703589371 (589h0m30.574672406s) [noise=true , address=145.79.227.8:4331 , piece_id=GFQBP7V5M4X67AASKKPBZ6QNGNULV62LVJWHQB7QVWXEPYFEQDQA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.703806304 (589h0m30.574889339s) [noise=true , address=145.79.227.39:4015 , piece_id=I6ZIVSJERWVISYX626AOLJKQHNBTCRSTDNPQL3FBDZCKR5SJHSWQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.880526099 (589h0m30.751609144s) [noise=true , address=77.90.31.16:4735 , piece_id=OBPAHXCEQ6JCQ2TTSJS6HVOKZIV3MWXM3I2ITOTNKFALIRJBY2NA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.965541162 (589h0m30.836624197s) [noise=true , address=77.90.31.17:4773 , piece_id=DQJCHBNPDKNCTFW2JNZWFBP5UCJRF57YE6QP4LSTHMBI7N5PA7XA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.990980616 (589h0m30.862063651s) [noise=true , address=77.90.31.19:4844 , piece_id=POZADDPXUYAHN4XSDUXVD6BSFHDDC6QPGSPQFV2TGT4775FT6DAA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:31.015210327 (589h0m30.886293372s) [noise=true , address=45.91.116.8:4048 , piece_id=UOP4NIRWNOPAHXLBIJ65HSIMRHFHE7RV2O7AY5BX6GDND3WM7BYA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:31.015233510 (589h0m30.886316555s) [seg_pos={0 0}, upload=516347]: Segment reached optimal threshold of 13 pieces. Dec 30 04:29:31.015437749 (589h0m30.886520784s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection Dec 30 04:29:31.015444632 (589h0m30.886527667s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015598556 (589h0m30.886681591s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:31.015604958 (589h0m30.886688003s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015679870 (589h0m30.886762915s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:31.015702764 (589h0m30.886785829s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015745716 (589h0m30.886828761s) [seg_pos={0 0}, upload=516347]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:29:31.015896483 (589h0m30.886979528s) [upload=516347]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:29:31.015906552 (589h0m30.886989587s) [upload=516347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:29:31.343639619 (589h0m31.214722664s) [upload=516347]: Done uploading segments... Dec 30 04:29:31.343662031 (589h0m31.214745076s) [upload=516347]: Upload finished. err: <nil> Duration: 6.00799868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX.] duration: 5.957525936s trace: https://jaeger.dev.storj.io/trace/4a5d023bb2664f91 FAIL (2025-12-30 04:30:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:29:28.434399217 (589h0m28.305482262s) [upload=516347]: Starting upload Dec 30 04:29:28.434489869 (589h0m28.305572914s) [upload=516347]: Done starting upload Dec 30 04:29:28.434617622 (589h0m28.305700657s) [upload=516347]: Uploading segments... Dec 30 04:29:28.434789019 (589h0m28.305872054s) [upload=516347]: Got next segment. Inline: false Dec 30 04:29:28.434836589 (589h0m28.305919624s) [upload=516347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:29:28.434845076 (589h0m28.305928121s) [upload=516347]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:29:28.682545798 (589h0m28.553628833s) [seg_pos={0 0}, upload=516347]: Begin upload segment... Dec 30 04:29:28.682596335 (589h0m28.55367938s) [seg_pos={0 0}, upload=516347]: Done begin upload segment. Dec 30 04:29:28.682619108 (589h0m28.553702153s) [noise=true , address=145.79.227.39:4015 , piece_id=I6ZIVSJERWVISYX626AOLJKQHNBTCRSTDNPQL3FBDZCKR5SJHSWQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.682630059 (589h0m28.553713094s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683215334 (589h0m28.554298369s) [noise=true , address=145.79.227.51:4411 , piece_id=F76JVREPOILNESWKXX7HDLP4Y2BSTD5D3HC7T7ZVGR6JXNRILH4A, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683544099 (589h0m28.554627134s) [noise=true , address=77.90.31.16:4735 , piece_id=OBPAHXCEQ6JCQ2TTSJS6HVOKZIV3MWXM3I2ITOTNKFALIRJBY2NA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683958839 (589h0m28.555041884s) [noise=true , address=145.79.227.52:4446 , piece_id=R6XOOOZBHBSLDZXYUVJ4EHNBDDY26YQOJ6FKJZQITOJCJOZ57VBQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.684284729 (589h0m28.555367774s) [noise=true , address=145.79.227.14:4837 , piece_id=GNQTXNCXEIYBHWLZCGMVV5HX67RVPKJQ37YZZYGZSQ5G4BO67ZVQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.684695121 (589h0m28.555778166s) [noise=true , address=77.90.31.17:4773 , piece_id=DQJCHBNPDKNCTFW2JNZWFBP5UCJRF57YE6QP4LSTHMBI7N5PA7XA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685041461 (589h0m28.556124496s) [noise=true , address=45.91.116.8:4048 , piece_id=UOP4NIRWNOPAHXLBIJ65HSIMRHFHE7RV2O7AY5BX6GDND3WM7BYA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685431825 (589h0m28.55651486s) [noise=true , address=145.79.227.8:4331 , piece_id=GFQBP7V5M4X67AASKKPBZ6QNGNULV62LVJWHQB7QVWXEPYFEQDQA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685760029 (589h0m28.556843074s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.686501010 (589h0m28.557584055s) [noise=true , address=145.79.227.11:4731 , piece_id=L3W4RC3BI6TTIP5VXKCY4IU2DNPHQUQ5A4YE4POHUNCDO2LSHS5Q, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.687010951 (589h0m28.558093996s) [noise=true , address=145.79.227.27:4689 , piece_id=65444YOGN66UYSRBELQCWW6I7V5BMT3TWSZX4HY5QK2G6GOR2K2A, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.687508569 (589h0m28.558591604s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688010144 (589h0m28.559093179s) [noise=true , address=145.79.227.16:4912 , piece_id=BDX7LJ5YUDKEKV467DWTQUEL7DHUGM3D7AX2XXPLTN2OXRIYNCHA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688470541 (589h0m28.559553576s) [noise=true , address=77.90.31.19:4844 , piece_id=POZADDPXUYAHN4XSDUXVD6BSFHDDC6QPGSPQFV2TGT4775FT6DAA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688966615 (589h0m28.56004966s) [noise=true , address=145.79.227.46:4601 , piece_id=56ED5TCCWZ5WHBCJQDTDHOC2WUPM6XJU2P24WDBG5BB2BTCZ5Q7Q, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:30.295075614 (589h0m30.166158649s) [upload=516347]: Next returned nil segment Dec 30 04:29:30.295090643 (589h0m30.166173698s) [upload=516347]: Waiting for error group managing segments... Dec 30 04:29:30.676109141 (589h0m30.547192176s) [noise=true , address=145.79.227.11:4731 , piece_id=L3W4RC3BI6TTIP5VXKCY4IU2DNPHQUQ5A4YE4POHUNCDO2LSHS5Q, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.680628126 (589h0m30.551711171s) [noise=true , address=145.79.227.46:4601 , piece_id=56ED5TCCWZ5WHBCJQDTDHOC2WUPM6XJU2P24WDBG5BB2BTCZ5Q7Q, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.685669807 (589h0m30.556752852s) [noise=true , address=145.79.227.14:4837 , piece_id=GNQTXNCXEIYBHWLZCGMVV5HX67RVPKJQ37YZZYGZSQ5G4BO67ZVQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.686564291 (589h0m30.557647336s) [noise=true , address=145.79.227.16:4912 , piece_id=BDX7LJ5YUDKEKV467DWTQUEL7DHUGM3D7AX2XXPLTN2OXRIYNCHA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.690018117 (589h0m30.561101162s) [noise=true , address=145.79.227.27:4689 , piece_id=65444YOGN66UYSRBELQCWW6I7V5BMT3TWSZX4HY5QK2G6GOR2K2A, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.699105572 (589h0m30.570188607s) [noise=true , address=145.79.227.51:4411 , piece_id=F76JVREPOILNESWKXX7HDLP4Y2BSTD5D3HC7T7ZVGR6JXNRILH4A, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.700519285 (589h0m30.57160232s) [noise=true , address=145.79.227.52:4446 , piece_id=R6XOOOZBHBSLDZXYUVJ4EHNBDDY26YQOJ6FKJZQITOJCJOZ57VBQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.703589371 (589h0m30.574672406s) [noise=true , address=145.79.227.8:4331 , piece_id=GFQBP7V5M4X67AASKKPBZ6QNGNULV62LVJWHQB7QVWXEPYFEQDQA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.703806304 (589h0m30.574889339s) [noise=true , address=145.79.227.39:4015 , piece_id=I6ZIVSJERWVISYX626AOLJKQHNBTCRSTDNPQL3FBDZCKR5SJHSWQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.880526099 (589h0m30.751609144s) [noise=true , address=77.90.31.16:4735 , piece_id=OBPAHXCEQ6JCQ2TTSJS6HVOKZIV3MWXM3I2ITOTNKFALIRJBY2NA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.965541162 (589h0m30.836624197s) [noise=true , address=77.90.31.17:4773 , piece_id=DQJCHBNPDKNCTFW2JNZWFBP5UCJRF57YE6QP4LSTHMBI7N5PA7XA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.990980616 (589h0m30.862063651s) [noise=true , address=77.90.31.19:4844 , piece_id=POZADDPXUYAHN4XSDUXVD6BSFHDDC6QPGSPQFV2TGT4775FT6DAA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:31.015210327 (589h0m30.886293372s) [noise=true , address=45.91.116.8:4048 , piece_id=UOP4NIRWNOPAHXLBIJ65HSIMRHFHE7RV2O7AY5BX6GDND3WM7BYA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:31.015233510 (589h0m30.886316555s) [seg_pos={0 0}, upload=516347]: Segment reached optimal threshold of 13 pieces. Dec 30 04:29:31.015437749 (589h0m30.886520784s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection Dec 30 04:29:31.015444632 (589h0m30.886527667s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015598556 (589h0m30.886681591s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:31.015604958 (589h0m30.886688003s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015679870 (589h0m30.886762915s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:31.015702764 (589h0m30.886785829s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015745716 (589h0m30.886828761s) [seg_pos={0 0}, upload=516347]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:29:31.015896483 (589h0m30.886979528s) [upload=516347]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:29:31.015906552 (589h0m30.886989587s) [upload=516347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:29:31.343639619 (589h0m31.214722664s) [upload=516347]: Done uploading segments... Dec 30 04:29:31.343662031 (589h0m31.214745076s) [upload=516347]: Upload finished. err: <nil> Duration: 5.957554159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X............X...................................] duration: 22.251210667s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:30:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 22.251237458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....XX] duration: 30.002394735s trace: https://jaeger.dev.storj.io/trace/4a5d023bb2664f91 FAIL (2025-12-30 04:29:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:29:28.434399217 (589h0m28.305482262s) [upload=516347]: Starting upload Dec 30 04:29:28.434489869 (589h0m28.305572914s) [upload=516347]: Done starting upload Dec 30 04:29:28.434617622 (589h0m28.305700657s) [upload=516347]: Uploading segments... Dec 30 04:29:28.434789019 (589h0m28.305872054s) [upload=516347]: Got next segment. Inline: false Dec 30 04:29:28.434836589 (589h0m28.305919624s) [upload=516347]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:29:28.434845076 (589h0m28.305928121s) [upload=516347]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:29:28.682545798 (589h0m28.553628833s) [seg_pos={0 0}, upload=516347]: Begin upload segment... Dec 30 04:29:28.682596335 (589h0m28.55367938s) [seg_pos={0 0}, upload=516347]: Done begin upload segment. Dec 30 04:29:28.682619108 (589h0m28.553702153s) [noise=true , address=145.79.227.39:4015 , piece_id=I6ZIVSJERWVISYX626AOLJKQHNBTCRSTDNPQL3FBDZCKR5SJHSWQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.682630059 (589h0m28.553713094s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683215334 (589h0m28.554298369s) [noise=true , address=145.79.227.51:4411 , piece_id=F76JVREPOILNESWKXX7HDLP4Y2BSTD5D3HC7T7ZVGR6JXNRILH4A, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683544099 (589h0m28.554627134s) [noise=true , address=77.90.31.16:4735 , piece_id=OBPAHXCEQ6JCQ2TTSJS6HVOKZIV3MWXM3I2ITOTNKFALIRJBY2NA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.683958839 (589h0m28.555041884s) [noise=true , address=145.79.227.52:4446 , piece_id=R6XOOOZBHBSLDZXYUVJ4EHNBDDY26YQOJ6FKJZQITOJCJOZ57VBQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.684284729 (589h0m28.555367774s) [noise=true , address=145.79.227.14:4837 , piece_id=GNQTXNCXEIYBHWLZCGMVV5HX67RVPKJQ37YZZYGZSQ5G4BO67ZVQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.684695121 (589h0m28.555778166s) [noise=true , address=77.90.31.17:4773 , piece_id=DQJCHBNPDKNCTFW2JNZWFBP5UCJRF57YE6QP4LSTHMBI7N5PA7XA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685041461 (589h0m28.556124496s) [noise=true , address=45.91.116.8:4048 , piece_id=UOP4NIRWNOPAHXLBIJ65HSIMRHFHE7RV2O7AY5BX6GDND3WM7BYA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685431825 (589h0m28.55651486s) [noise=true , address=145.79.227.8:4331 , piece_id=GFQBP7V5M4X67AASKKPBZ6QNGNULV62LVJWHQB7QVWXEPYFEQDQA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.685760029 (589h0m28.556843074s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.686501010 (589h0m28.557584055s) [noise=true , address=145.79.227.11:4731 , piece_id=L3W4RC3BI6TTIP5VXKCY4IU2DNPHQUQ5A4YE4POHUNCDO2LSHS5Q, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.687010951 (589h0m28.558093996s) [noise=true , address=145.79.227.27:4689 , piece_id=65444YOGN66UYSRBELQCWW6I7V5BMT3TWSZX4HY5QK2G6GOR2K2A, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.687508569 (589h0m28.558591604s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688010144 (589h0m28.559093179s) [noise=true , address=145.79.227.16:4912 , piece_id=BDX7LJ5YUDKEKV467DWTQUEL7DHUGM3D7AX2XXPLTN2OXRIYNCHA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688470541 (589h0m28.559553576s) [noise=true , address=77.90.31.19:4844 , piece_id=POZADDPXUYAHN4XSDUXVD6BSFHDDC6QPGSPQFV2TGT4775FT6DAA, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:28.688966615 (589h0m28.56004966s) [noise=true , address=145.79.227.46:4601 , piece_id=56ED5TCCWZ5WHBCJQDTDHOC2WUPM6XJU2P24WDBG5BB2BTCZ5Q7Q, seg_pos={0 0}, upload=516347]: Uploading piece... Dec 30 04:29:30.295075614 (589h0m30.166158649s) [upload=516347]: Next returned nil segment Dec 30 04:29:30.295090643 (589h0m30.166173698s) [upload=516347]: Waiting for error group managing segments... Dec 30 04:29:30.676109141 (589h0m30.547192176s) [noise=true , address=145.79.227.11:4731 , piece_id=L3W4RC3BI6TTIP5VXKCY4IU2DNPHQUQ5A4YE4POHUNCDO2LSHS5Q, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.680628126 (589h0m30.551711171s) [noise=true , address=145.79.227.46:4601 , piece_id=56ED5TCCWZ5WHBCJQDTDHOC2WUPM6XJU2P24WDBG5BB2BTCZ5Q7Q, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.685669807 (589h0m30.556752852s) [noise=true , address=145.79.227.14:4837 , piece_id=GNQTXNCXEIYBHWLZCGMVV5HX67RVPKJQ37YZZYGZSQ5G4BO67ZVQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.686564291 (589h0m30.557647336s) [noise=true , address=145.79.227.16:4912 , piece_id=BDX7LJ5YUDKEKV467DWTQUEL7DHUGM3D7AX2XXPLTN2OXRIYNCHA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.690018117 (589h0m30.561101162s) [noise=true , address=145.79.227.27:4689 , piece_id=65444YOGN66UYSRBELQCWW6I7V5BMT3TWSZX4HY5QK2G6GOR2K2A, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.699105572 (589h0m30.570188607s) [noise=true , address=145.79.227.51:4411 , piece_id=F76JVREPOILNESWKXX7HDLP4Y2BSTD5D3HC7T7ZVGR6JXNRILH4A, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.700519285 (589h0m30.57160232s) [noise=true , address=145.79.227.52:4446 , piece_id=R6XOOOZBHBSLDZXYUVJ4EHNBDDY26YQOJ6FKJZQITOJCJOZ57VBQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.703589371 (589h0m30.574672406s) [noise=true , address=145.79.227.8:4331 , piece_id=GFQBP7V5M4X67AASKKPBZ6QNGNULV62LVJWHQB7QVWXEPYFEQDQA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.703806304 (589h0m30.574889339s) [noise=true , address=145.79.227.39:4015 , piece_id=I6ZIVSJERWVISYX626AOLJKQHNBTCRSTDNPQL3FBDZCKR5SJHSWQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.880526099 (589h0m30.751609144s) [noise=true , address=77.90.31.16:4735 , piece_id=OBPAHXCEQ6JCQ2TTSJS6HVOKZIV3MWXM3I2ITOTNKFALIRJBY2NA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.965541162 (589h0m30.836624197s) [noise=true , address=77.90.31.17:4773 , piece_id=DQJCHBNPDKNCTFW2JNZWFBP5UCJRF57YE6QP4LSTHMBI7N5PA7XA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:30.990980616 (589h0m30.862063651s) [noise=true , address=77.90.31.19:4844 , piece_id=POZADDPXUYAHN4XSDUXVD6BSFHDDC6QPGSPQFV2TGT4775FT6DAA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:31.015210327 (589h0m30.886293372s) [noise=true , address=45.91.116.8:4048 , piece_id=UOP4NIRWNOPAHXLBIJ65HSIMRHFHE7RV2O7AY5BX6GDND3WM7BYA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: <nil> Dec 30 04:29:31.015233510 (589h0m30.886316555s) [seg_pos={0 0}, upload=516347]: Segment reached optimal threshold of 13 pieces. Dec 30 04:29:31.015437749 (589h0m30.886520784s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44914->77.90.31.13:4626: use of closed network connection Dec 30 04:29:31.015444632 (589h0m30.886527667s) [noise=true , address=77.90.31.13:4626 , piece_id=OUP45BV3PL743ZQN6KRB3DBBUCF7RGELEEIKF6DXP34SBZVK25YQ, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015598556 (589h0m30.886681591s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:31.015604958 (589h0m30.886688003s) [noise=true , address=45.91.116.14:4192 , piece_id=FHP6IX3KUZDNMVSSIJKNBT52D34A2EEUF2OKFCDUKS76SPY5WSGQ, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015679870 (589h0m30.886762915s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:31.015702764 (589h0m30.886785829s) [noise=true , address=45.91.116.9:4263 , piece_id=3D6I5ENNKC6JI7CVE2755PKNV5KEPTNR2KQQIPALCQEDDZ7A5ZNA, seg_pos={0 0}, upload=516347]: Long tail cancellation due to optimal threshold Dec 30 04:29:31.015745716 (589h0m30.886828761s) [seg_pos={0 0}, upload=516347]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:29:31.015896483 (589h0m30.886979528s) [upload=516347]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:29:31.015906552 (589h0m30.886989587s) [upload=516347]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:29:31.343639619 (589h0m31.214722664s) [upload=516347]: Done uploading segments... Dec 30 04:29:31.343662031 (589h0m31.214745076s) [upload=516347]: Upload finished. err: <nil> Duration: 30.002409953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X............X..................................] duration: 45.387736034s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:29:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 45.387770019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......X..XXX............XX..X.....X..............X..........] duration: 3.540239103s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:29:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540268518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....X] duration: 30.003092699s trace: https://jaeger.dev.storj.io/trace/1c69dd157bee0bca FAIL (2025-12-30 04:29:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:28:58.193093984 (588h59m58.064177029s) [upload=516339]: Starting upload Dec 30 04:28:58.193174918 (588h59m58.064257953s) [upload=516339]: Done starting upload Dec 30 04:28:58.193275530 (588h59m58.064358575s) [upload=516339]: Uploading segments... Dec 30 04:28:58.193399185 (588h59m58.06448222s) [upload=516339]: Got next segment. Inline: false Dec 30 04:28:58.193441065 (588h59m58.0645241s) [upload=516339]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:28:58.193446245 (588h59m58.06452928s) [upload=516339]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:28:58.440848559 (588h59m58.311931594s) [seg_pos={0 0}, upload=516339]: Begin upload segment... Dec 30 04:28:58.440895258 (588h59m58.311978293s) [seg_pos={0 0}, upload=516339]: Done begin upload segment. Dec 30 04:28:58.440919464 (588h59m58.312002499s) [noise=true , address=145.79.227.11:4729 , piece_id=NP2LCREKUEF76SY4OFVDJWRDPT4YA2ZOOO4BE5C4AR427QHL36IQ, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.440940534 (588h59m58.312023569s) [noise=true , address=45.91.116.11:4119 , piece_id=FJCGCLPTGEWGHTEXZUTZE2FEAQTCPRR3GOLZGRVXNLTIYBBOTNRQ, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.441459642 (588h59m58.312542687s) [noise=true , address=45.91.116.6:4975 , piece_id=CLUU4SHZSR6N6HOTNUXC5PECEE6RTKE4IZZILJSYGGWF4MLGM7PA, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.441961328 (588h59m58.313044373s) [noise=true , address=145.79.227.14:4840 , piece_id=54MSIWFWFCXR3VJURCEDLEL3DRK3IRHGTFMVNYLBK4LBU2XBOQ4Q, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.442529710 (588h59m58.313612755s) [noise=true , address=145.79.227.17:4970 , piece_id=RGH7YZPQEMU7SEJ5VMGBSUS74DGY4HQE6UNGZP567RCHSDWUFG3A, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.443652278 (588h59m58.314735313s) [noise=true , address=77.90.31.16:4736 , piece_id=33KGZCFSEKMR55OUV4AXNOMTHASMEE2R5JP2JLW5O7YJJ5CXXBYA, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.444164664 (588h59m58.315247699s) [noise=true , address=145.79.227.46:4601 , piece_id=4WBO4T4UWNSKZVB3IK46KET2TUJAUAQXWLBVFOTH6F2XZFQOB3TA, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.444655117 (588h59m58.315738162s) [noise=true , address=45.91.116.12:4225 , piece_id=JDUHWLUK37ABC5FKNAGMF2V57KAIG6FH6KHZYATICFQQEBWNE4FA, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.445191970 (588h59m58.316275015s) [noise=true , address=145.79.227.48:4302 , piece_id=5TWXI2IUF2OA4FNTOZM7XB4BINWZUDDHVOUP4O7652ZOKLLQGMFQ, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.445750193 (588h59m58.316833228s) [noise=true , address=145.79.227.38:4977 , piece_id=JUH273IF7X4MQAQCFIN7KOWT6SZWGVVN74PXGHAYGJJ2ZGXF52JA, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.446383058 (588h59m58.317466093s) [noise=true , address=145.79.227.51:4409 , piece_id=ZGSBXZFOWJ25INBCBHF2F6FOJ4YHOB364ADMMJT37BJUFLCWFHTQ, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.448078707 (588h59m58.319161772s) [noise=true , address=145.79.227.23:4545 , piece_id=QIHJJTZ3BDZYJ2XSVVLCTQR7M2P6EAFEKKX4LD4WZAXINVASDS6A, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.449716275 (588h59m58.32079932s) [noise=true , address=45.91.116.14:4192 , piece_id=YU4U5UPJUWK5WBNGNZWFP57BFG4Q3VYUIAC6ZZWWJZC7PEYPOMCQ, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.451934479 (588h59m58.323017514s) [noise=true , address=145.79.227.37:4567 , piece_id=XXBYQUMWJ24IXQBAGK344ED2X3B7KI4NDDBLLWZQEYSGGLOKQCQQ, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.452410767 (588h59m58.323493802s) [noise=true , address=45.91.116.4:4937 , piece_id=R427DIGT7D55FUG6EWOV7WS3EOBQ323CN7YL2IJ2ZO5WZPDJZARQ, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:28:58.453698809 (588h59m58.324781844s) [noise=true , address=77.90.31.18:4808 , piece_id=H3PAMLQJCUIR2GUJVJ4EETGCBLI7W4FMMTJQG6XZJVEXMUEA4JMA, seg_pos={0 0}, upload=516339]: Uploading piece... Dec 30 04:29:00.039135352 (588h59m59.910218387s) [upload=516339]: Next returned nil segment Dec 30 04:29:00.039147866 (588h59m59.910230901s) [upload=516339]: Waiting for error group managing segments... Dec 30 04:29:00.415739112 (589h0m0.286822157s) [noise=true , address=145.79.227.17:4970 , piece_id=RGH7YZPQEMU7SEJ5VMGBSUS74DGY4HQE6UNGZP567RCHSDWUFG3A, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.425573971 (589h0m0.296657016s) [noise=true , address=145.79.227.51:4409 , piece_id=ZGSBXZFOWJ25INBCBHF2F6FOJ4YHOB364ADMMJT37BJUFLCWFHTQ, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.429238780 (589h0m0.300321815s) [noise=true , address=145.79.227.14:4840 , piece_id=54MSIWFWFCXR3VJURCEDLEL3DRK3IRHGTFMVNYLBK4LBU2XBOQ4Q, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.441209105 (589h0m0.31229215s) [noise=true , address=145.79.227.48:4302 , piece_id=5TWXI2IUF2OA4FNTOZM7XB4BINWZUDDHVOUP4O7652ZOKLLQGMFQ, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.442789404 (589h0m0.313872449s) [noise=true , address=145.79.227.38:4977 , piece_id=JUH273IF7X4MQAQCFIN7KOWT6SZWGVVN74PXGHAYGJJ2ZGXF52JA, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.442906247 (589h0m0.313989292s) [noise=true , address=145.79.227.23:4545 , piece_id=QIHJJTZ3BDZYJ2XSVVLCTQR7M2P6EAFEKKX4LD4WZAXINVASDS6A, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.449361489 (589h0m0.320444534s) [noise=true , address=145.79.227.11:4729 , piece_id=NP2LCREKUEF76SY4OFVDJWRDPT4YA2ZOOO4BE5C4AR427QHL36IQ, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.461891671 (589h0m0.332974716s) [noise=true , address=145.79.227.37:4567 , piece_id=XXBYQUMWJ24IXQBAGK344ED2X3B7KI4NDDBLLWZQEYSGGLOKQCQQ, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.469491042 (589h0m0.340574077s) [noise=true , address=145.79.227.46:4601 , piece_id=4WBO4T4UWNSKZVB3IK46KET2TUJAUAQXWLBVFOTH6F2XZFQOB3TA, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.621016248 (589h0m0.492099283s) [noise=true , address=77.90.31.16:4736 , piece_id=33KGZCFSEKMR55OUV4AXNOMTHASMEE2R5JP2JLW5O7YJJ5CXXBYA, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.681885702 (589h0m0.552968737s) [noise=true , address=45.91.116.14:4192 , piece_id=YU4U5UPJUWK5WBNGNZWFP57BFG4Q3VYUIAC6ZZWWJZC7PEYPOMCQ, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.708188349 (589h0m0.579271394s) [noise=true , address=45.91.116.12:4225 , piece_id=JDUHWLUK37ABC5FKNAGMF2V57KAIG6FH6KHZYATICFQQEBWNE4FA, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.803413024 (589h0m0.674496069s) [noise=true , address=45.91.116.6:4975 , piece_id=CLUU4SHZSR6N6HOTNUXC5PECEE6RTKE4IZZILJSYGGWF4MLGM7PA, seg_pos={0 0}, upload=516339]: Done uploading piece. err: <nil> Dec 30 04:29:00.803502164 (589h0m0.674585199s) [seg_pos={0 0}, upload=516339]: Segment reached optimal threshold of 13 pieces. Dec 30 04:29:00.803696935 (589h0m0.67477998s) [noise=true , address=77.90.31.18:4808 , piece_id=H3PAMLQJCUIR2GUJVJ4EETGCBLI7W4FMMTJQG6XZJVEXMUEA4JMA, seg_pos={0 0}, upload=516339]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:00.803705120 (589h0m0.674788166s) [noise=true , address=77.90.31.18:4808 , piece_id=H3PAMLQJCUIR2GUJVJ4EETGCBLI7W4FMMTJQG6XZJVEXMUEA4JMA, seg_pos={0 0}, upload=516339]: Long tail cancellation due to optimal threshold Dec 30 04:29:00.803848794 (589h0m0.674931839s) [noise=true , address=45.91.116.11:4119 , piece_id=FJCGCLPTGEWGHTEXZUTZE2FEAQTCPRR3GOLZGRVXNLTIYBBOTNRQ, seg_pos={0 0}, upload=516339]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:00.803879843 (589h0m0.674962878s) [noise=true , address=45.91.116.11:4119 , piece_id=FJCGCLPTGEWGHTEXZUTZE2FEAQTCPRR3GOLZGRVXNLTIYBBOTNRQ, seg_pos={0 0}, upload=516339]: Long tail cancellation due to optimal threshold Dec 30 04:29:00.803920340 (589h0m0.675003375s) [noise=true , address=45.91.116.4:4937 , piece_id=R427DIGT7D55FUG6EWOV7WS3EOBQ323CN7YL2IJ2ZO5WZPDJZARQ, seg_pos={0 0}, upload=516339]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 30 04:29:00.803928225 (589h0m0.67501126s) [noise=true , address=45.91.116.4:4937 , piece_id=R427DIGT7D55FUG6EWOV7WS3EOBQ323CN7YL2IJ2ZO5WZPDJZARQ, seg_pos={0 0}, upload=516339]: Long tail cancellation due to optimal threshold Dec 30 04:29:00.803958052 (589h0m0.675041087s) [seg_pos={0 0}, upload=516339]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:29:00.804080806 (589h0m0.675163841s) [upload=516339]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:29:00.804085956 (589h0m0.675169001s) [upload=516339]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:29:01.111236137 (589h0m0.982319172s) [upload=516339]: Done uploading segments... Dec 30 04:29:01.111253961 (589h0m0.982337006s) [upload=516339]: Upload finished. err: <nil> Duration: 30.003118487s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......X..XXX............XX..X.....X..............X.........] duration: 3.427534918s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:29:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427567672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X............X.................................] duration: 41.432463183s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:29:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 41.432491577s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X....] duration: 5.991556029s trace: https://jaeger.dev.storj.io/trace/1f53df790d5e45b5 FAIL (2025-12-30 04:28:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA) context deadline exceeded ======LOGS====== 0 04:26:40.998161535 (588h57m40.86924457s) [upload=516303]: Starting upload Dec 30 04:26:40.998354102 (588h57m40.869437137s) [upload=516303]: Done starting upload Dec 30 04:26:40.998436198 (588h57m40.869519243s) [upload=516303]: Uploading segments... Dec 30 04:26:40.998600461 (588h57m40.869683496s) [upload=516303]: Got next segment. Inline: false Dec 30 04:26:40.998661446 (588h57m40.869744491s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:26:40.998668561 (588h57m40.869751596s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:26:41.244820835 (588h57m41.11590388s) [seg_pos={0 0}, upload=516303]: Begin upload segment... Dec 30 04:26:41.244892351 (588h57m41.115975386s) [seg_pos={0 0}, upload=516303]: Done begin upload segment. Dec 30 04:26:41.245192443 (588h57m41.116275488s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.245233631 (588h57m41.116316676s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246038263 (588h57m41.117121308s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246987130 (588h57m41.118070175s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.248784212 (588h57m41.119867247s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.251774927 (588h57m41.122857972s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.252044541 (588h57m41.123127576s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255045254 (588h57m41.126128299s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255571547 (588h57m41.126654582s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257010908 (588h57m41.128093943s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257520969 (588h57m41.128604014s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.260276146 (588h57m41.131359181s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.261793726 (588h57m41.132876771s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.263003890 (588h57m41.134086935s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.269263972 (588h57m41.140347017s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.270622218 (588h57m41.141705263s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:42.967182180 (588h57m42.838265215s) [upload=516303]: Next returned nil segment Dec 30 04:26:42.967206306 (588h57m42.838289341s) [upload=516303]: Waiting for error group managing segments... Dec 30 04:26:43.247027766 (588h57m43.118110801s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.247983896 (588h57m43.119066931s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250482314 (588h57m43.121565349s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250747750 (588h57m43.121830785s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.255405058 (588h57m43.126488093s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.256707087 (588h57m43.127790122s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.262786014 (588h57m43.133869049s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.268122807 (588h57m43.139205842s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.283793989 (588h57m43.154877034s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.284748777 (588h57m43.155831812s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.489137372 (588h57m43.360220407s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.510976179 (588h57m43.382059224s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543934010 (588h57m43.415017055s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543955230 (588h57m43.415038265s) [seg_pos={0 0}, upload=516303]: Segment reached optimal threshold of 13 pieces. Dec 30 04:26:43.544174548 (588h57m43.415257593s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544180860 (588h57m43.415263905s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544407642 (588h57m43.415490677s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection Dec 30 04:26:43.544414335 (588h57m43.41549738s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544466544 (588h57m43.415549599s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544487103 (588h57m43.415570138s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544528522 (588h57m43.415611557s) [seg_pos={0 0}, upload=516303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:26:43.544605859 (588h57m43.415688894s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:26:43.544610407 (588h57m43.415693452s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:26:43.831290249 (588h57m43.702373294s) [upload=516303]: Done uploading segments... Dec 30 04:26:43.831318373 (588h57m43.702401448s) [upload=516303]: Upload finished. err: <nil> Duration: 5.991587388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X...] duration: 6.005269582s trace: https://jaeger.dev.storj.io/trace/1f53df790d5e45b5 FAIL (2025-12-30 04:28:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA) context deadline exceeded ======LOGS====== 0 04:26:40.998161535 (588h57m40.86924457s) [upload=516303]: Starting upload Dec 30 04:26:40.998354102 (588h57m40.869437137s) [upload=516303]: Done starting upload Dec 30 04:26:40.998436198 (588h57m40.869519243s) [upload=516303]: Uploading segments... Dec 30 04:26:40.998600461 (588h57m40.869683496s) [upload=516303]: Got next segment. Inline: false Dec 30 04:26:40.998661446 (588h57m40.869744491s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:26:40.998668561 (588h57m40.869751596s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:26:41.244820835 (588h57m41.11590388s) [seg_pos={0 0}, upload=516303]: Begin upload segment... Dec 30 04:26:41.244892351 (588h57m41.115975386s) [seg_pos={0 0}, upload=516303]: Done begin upload segment. Dec 30 04:26:41.245192443 (588h57m41.116275488s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.245233631 (588h57m41.116316676s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246038263 (588h57m41.117121308s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246987130 (588h57m41.118070175s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.248784212 (588h57m41.119867247s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.251774927 (588h57m41.122857972s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.252044541 (588h57m41.123127576s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255045254 (588h57m41.126128299s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255571547 (588h57m41.126654582s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257010908 (588h57m41.128093943s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257520969 (588h57m41.128604014s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.260276146 (588h57m41.131359181s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.261793726 (588h57m41.132876771s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.263003890 (588h57m41.134086935s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.269263972 (588h57m41.140347017s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.270622218 (588h57m41.141705263s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:42.967182180 (588h57m42.838265215s) [upload=516303]: Next returned nil segment Dec 30 04:26:42.967206306 (588h57m42.838289341s) [upload=516303]: Waiting for error group managing segments... Dec 30 04:26:43.247027766 (588h57m43.118110801s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.247983896 (588h57m43.119066931s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250482314 (588h57m43.121565349s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250747750 (588h57m43.121830785s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.255405058 (588h57m43.126488093s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.256707087 (588h57m43.127790122s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.262786014 (588h57m43.133869049s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.268122807 (588h57m43.139205842s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.283793989 (588h57m43.154877034s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.284748777 (588h57m43.155831812s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.489137372 (588h57m43.360220407s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.510976179 (588h57m43.382059224s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543934010 (588h57m43.415017055s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543955230 (588h57m43.415038265s) [seg_pos={0 0}, upload=516303]: Segment reached optimal threshold of 13 pieces. Dec 30 04:26:43.544174548 (588h57m43.415257593s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544180860 (588h57m43.415263905s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544407642 (588h57m43.415490677s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection Dec 30 04:26:43.544414335 (588h57m43.41549738s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544466544 (588h57m43.415549599s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544487103 (588h57m43.415570138s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544528522 (588h57m43.415611557s) [seg_pos={0 0}, upload=516303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:26:43.544605859 (588h57m43.415688894s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:26:43.544610407 (588h57m43.415693452s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:26:43.831290249 (588h57m43.702373294s) [upload=516303]: Done uploading segments... Dec 30 04:26:43.831318373 (588h57m43.702401448s) [upload=516303]: Upload finished. err: <nil> Duration: 6.005298938s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [........X..XXX............XX..X.....X..............X........] duration: 3.377964967s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:27:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.378018168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X..] duration: 5.653941304s trace: https://jaeger.dev.storj.io/trace/1f53df790d5e45b5 FAIL (2025-12-30 04:27:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA) context deadline exceeded ======LOGS====== 0 04:26:40.998161535 (588h57m40.86924457s) [upload=516303]: Starting upload Dec 30 04:26:40.998354102 (588h57m40.869437137s) [upload=516303]: Done starting upload Dec 30 04:26:40.998436198 (588h57m40.869519243s) [upload=516303]: Uploading segments... Dec 30 04:26:40.998600461 (588h57m40.869683496s) [upload=516303]: Got next segment. Inline: false Dec 30 04:26:40.998661446 (588h57m40.869744491s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:26:40.998668561 (588h57m40.869751596s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:26:41.244820835 (588h57m41.11590388s) [seg_pos={0 0}, upload=516303]: Begin upload segment... Dec 30 04:26:41.244892351 (588h57m41.115975386s) [seg_pos={0 0}, upload=516303]: Done begin upload segment. Dec 30 04:26:41.245192443 (588h57m41.116275488s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.245233631 (588h57m41.116316676s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246038263 (588h57m41.117121308s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246987130 (588h57m41.118070175s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.248784212 (588h57m41.119867247s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.251774927 (588h57m41.122857972s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.252044541 (588h57m41.123127576s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255045254 (588h57m41.126128299s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255571547 (588h57m41.126654582s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257010908 (588h57m41.128093943s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257520969 (588h57m41.128604014s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.260276146 (588h57m41.131359181s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.261793726 (588h57m41.132876771s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.263003890 (588h57m41.134086935s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.269263972 (588h57m41.140347017s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.270622218 (588h57m41.141705263s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:42.967182180 (588h57m42.838265215s) [upload=516303]: Next returned nil segment Dec 30 04:26:42.967206306 (588h57m42.838289341s) [upload=516303]: Waiting for error group managing segments... Dec 30 04:26:43.247027766 (588h57m43.118110801s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.247983896 (588h57m43.119066931s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250482314 (588h57m43.121565349s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250747750 (588h57m43.121830785s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.255405058 (588h57m43.126488093s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.256707087 (588h57m43.127790122s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.262786014 (588h57m43.133869049s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.268122807 (588h57m43.139205842s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.283793989 (588h57m43.154877034s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.284748777 (588h57m43.155831812s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.489137372 (588h57m43.360220407s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.510976179 (588h57m43.382059224s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543934010 (588h57m43.415017055s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543955230 (588h57m43.415038265s) [seg_pos={0 0}, upload=516303]: Segment reached optimal threshold of 13 pieces. Dec 30 04:26:43.544174548 (588h57m43.415257593s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544180860 (588h57m43.415263905s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544407642 (588h57m43.415490677s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection Dec 30 04:26:43.544414335 (588h57m43.41549738s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544466544 (588h57m43.415549599s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544487103 (588h57m43.415570138s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544528522 (588h57m43.415611557s) [seg_pos={0 0}, upload=516303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:26:43.544605859 (588h57m43.415688894s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:26:43.544610407 (588h57m43.415693452s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:26:43.831290249 (588h57m43.702373294s) [upload=516303]: Done uploading segments... Dec 30 04:26:43.831318373 (588h57m43.702401448s) [upload=516303]: Upload finished. err: <nil> Duration: 5.653966111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X............X................................] duration: 30.477484193s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:27:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 30.477499361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X.] duration: 5.974037007s trace: https://jaeger.dev.storj.io/trace/1f53df790d5e45b5 FAIL (2025-12-30 04:27:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA) context deadline exceeded ======LOGS====== 0 04:26:40.998161535 (588h57m40.86924457s) [upload=516303]: Starting upload Dec 30 04:26:40.998354102 (588h57m40.869437137s) [upload=516303]: Done starting upload Dec 30 04:26:40.998436198 (588h57m40.869519243s) [upload=516303]: Uploading segments... Dec 30 04:26:40.998600461 (588h57m40.869683496s) [upload=516303]: Got next segment. Inline: false Dec 30 04:26:40.998661446 (588h57m40.869744491s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:26:40.998668561 (588h57m40.869751596s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:26:41.244820835 (588h57m41.11590388s) [seg_pos={0 0}, upload=516303]: Begin upload segment... Dec 30 04:26:41.244892351 (588h57m41.115975386s) [seg_pos={0 0}, upload=516303]: Done begin upload segment. Dec 30 04:26:41.245192443 (588h57m41.116275488s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.245233631 (588h57m41.116316676s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246038263 (588h57m41.117121308s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246987130 (588h57m41.118070175s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.248784212 (588h57m41.119867247s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.251774927 (588h57m41.122857972s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.252044541 (588h57m41.123127576s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255045254 (588h57m41.126128299s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255571547 (588h57m41.126654582s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257010908 (588h57m41.128093943s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257520969 (588h57m41.128604014s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.260276146 (588h57m41.131359181s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.261793726 (588h57m41.132876771s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.263003890 (588h57m41.134086935s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.269263972 (588h57m41.140347017s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.270622218 (588h57m41.141705263s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:42.967182180 (588h57m42.838265215s) [upload=516303]: Next returned nil segment Dec 30 04:26:42.967206306 (588h57m42.838289341s) [upload=516303]: Waiting for error group managing segments... Dec 30 04:26:43.247027766 (588h57m43.118110801s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.247983896 (588h57m43.119066931s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250482314 (588h57m43.121565349s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250747750 (588h57m43.121830785s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.255405058 (588h57m43.126488093s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.256707087 (588h57m43.127790122s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.262786014 (588h57m43.133869049s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.268122807 (588h57m43.139205842s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.283793989 (588h57m43.154877034s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.284748777 (588h57m43.155831812s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.489137372 (588h57m43.360220407s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.510976179 (588h57m43.382059224s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543934010 (588h57m43.415017055s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543955230 (588h57m43.415038265s) [seg_pos={0 0}, upload=516303]: Segment reached optimal threshold of 13 pieces. Dec 30 04:26:43.544174548 (588h57m43.415257593s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544180860 (588h57m43.415263905s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544407642 (588h57m43.415490677s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection Dec 30 04:26:43.544414335 (588h57m43.41549738s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544466544 (588h57m43.415549599s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544487103 (588h57m43.415570138s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544528522 (588h57m43.415611557s) [seg_pos={0 0}, upload=516303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:26:43.544605859 (588h57m43.415688894s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:26:43.544610407 (588h57m43.415693452s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:26:43.831290249 (588h57m43.702373294s) [upload=516303]: Done uploading segments... Dec 30 04:26:43.831318373 (588h57m43.702401448s) [upload=516303]: Upload finished. err: <nil> Duration: 5.974068265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X............X...............................] duration: 22.161952597s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:27:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 22.161982293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..X] duration: 30.003742155s trace: https://jaeger.dev.storj.io/trace/1f53df790d5e45b5 FAIL (2025-12-30 04:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA) context deadline exceeded ======LOGS====== 0 04:26:40.998161535 (588h57m40.86924457s) [upload=516303]: Starting upload Dec 30 04:26:40.998354102 (588h57m40.869437137s) [upload=516303]: Done starting upload Dec 30 04:26:40.998436198 (588h57m40.869519243s) [upload=516303]: Uploading segments... Dec 30 04:26:40.998600461 (588h57m40.869683496s) [upload=516303]: Got next segment. Inline: false Dec 30 04:26:40.998661446 (588h57m40.869744491s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:26:40.998668561 (588h57m40.869751596s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:26:41.244820835 (588h57m41.11590388s) [seg_pos={0 0}, upload=516303]: Begin upload segment... Dec 30 04:26:41.244892351 (588h57m41.115975386s) [seg_pos={0 0}, upload=516303]: Done begin upload segment. Dec 30 04:26:41.245192443 (588h57m41.116275488s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.245233631 (588h57m41.116316676s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246038263 (588h57m41.117121308s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.246987130 (588h57m41.118070175s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.248784212 (588h57m41.119867247s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.251774927 (588h57m41.122857972s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.252044541 (588h57m41.123127576s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255045254 (588h57m41.126128299s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.255571547 (588h57m41.126654582s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257010908 (588h57m41.128093943s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.257520969 (588h57m41.128604014s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.260276146 (588h57m41.131359181s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.261793726 (588h57m41.132876771s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.263003890 (588h57m41.134086935s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.269263972 (588h57m41.140347017s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:41.270622218 (588h57m41.141705263s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Uploading piece... Dec 30 04:26:42.967182180 (588h57m42.838265215s) [upload=516303]: Next returned nil segment Dec 30 04:26:42.967206306 (588h57m42.838289341s) [upload=516303]: Waiting for error group managing segments... Dec 30 04:26:43.247027766 (588h57m43.118110801s) [noise=true , address=145.79.227.5:4222 , piece_id=5MLYBKLGU6XJPR3QDGKKNI2QJA4I3SND3MRNG7B633FRDJNUYUQA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.247983896 (588h57m43.119066931s) [noise=true , address=145.79.227.32:4836 , piece_id=KFHKFWD2RJMOHRGC3LLCWAUD2NOZEIODYDGYPGUGIEHR2M5EAIHQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250482314 (588h57m43.121565349s) [noise=true , address=145.79.227.28:4725 , piece_id=KGYWGI4IVUBRNYESYDQKLKLBP7NZJASBLP7L4YM2Y3ES3PM2IP7Q, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.250747750 (588h57m43.121830785s) [noise=true , address=145.79.227.45:4229 , piece_id=7X2S4LK767CWCTSK6LIJONEMH3YY4UBTCJ4S5XWNKKMGQKRUDNIA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.255405058 (588h57m43.126488093s) [noise=true , address=145.79.227.36:4518 , piece_id=BS7PWYJ3AMQEPUXMCXYF7KUFBUOBY6UUAWV3IRJZV5NJTORP2ZVQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.256707087 (588h57m43.127790122s) [noise=true , address=145.79.227.10:4401 , piece_id=UKV3LQKIFC2AV3YT2P3WVSHA56ABWA7QEL7NRJW4ET2RK7AMUR3A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.262786014 (588h57m43.133869049s) [noise=true , address=145.79.227.12:4767 , piece_id=U5BVIRDFN7KLT2ODFP4HPYEGZXWYIIT3BAWVGLWRL5UZFOZRTIEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.268122807 (588h57m43.139205842s) [noise=true , address=145.79.227.47:4265 , piece_id=OLSFWB5B7WBX2WO32BQMXJ2KCER6WMSTWM4CIELPNQIZFTCLYQNA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.283793989 (588h57m43.154877034s) [noise=true , address=145.79.227.51:4410 , piece_id=KD4YC2JAUH4G5MLFL4LA6PVYH6ZPCV5HS7YG2ZTGFX2L6YYLT5YA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.284748777 (588h57m43.155831812s) [noise=true , address=145.79.227.37:4565 , piece_id=TFXMF4YKYJGWF2GETXTPCF5NB4THDO6YONATNG4NJ7Q7D7QET45A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.489137372 (588h57m43.360220407s) [noise=true , address=77.90.31.17:4773 , piece_id=QWY3HMOLSMDQCM5MW6EJY5DXRSI7BNU23PGSQECAQXDLEC6VEL6A, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.510976179 (588h57m43.382059224s) [noise=true , address=45.91.116.11:4118 , piece_id=5AL5IODKFO7JEBXUIIUI66DI3DA6JGS7PNTBXOJBITWB7CTADAHA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543934010 (588h57m43.415017055s) [noise=true , address=45.91.116.13:4154 , piece_id=4PWOPKZ7BH5EUS6BHFEV7AAMSBIAGJGGSV5LEHHIJAIVYDP43ZEA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: <nil> Dec 30 04:26:43.543955230 (588h57m43.415038265s) [seg_pos={0 0}, upload=516303]: Segment reached optimal threshold of 13 pieces. Dec 30 04:26:43.544174548 (588h57m43.415257593s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544180860 (588h57m43.415263905s) [noise=true , address=45.91.116.10:4083 , piece_id=QOJT5OEKF5HWHL5Y6PNO7AJILZ2MJUZNG7NXPYLSDHZQIHT426LA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544407642 (588h57m43.415490677s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55460->45.91.116.8:4047: use of closed network connection Dec 30 04:26:43.544414335 (588h57m43.41549738s) [noise=true , address=45.91.116.8:4047 , piece_id=XA7LEN7N355VWYWVBZV54AI4DQLH7TFVINVIJML4JU6GTMIMLRQQ, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544466544 (588h57m43.415549599s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 04:26:43.544487103 (588h57m43.415570138s) [noise=true , address=145.79.227.30:4762 , piece_id=KDU74RCL5RTDXVLQ26IKUE4PWJMP5NUE2IMZE6WLVFI6PORNEDOA, seg_pos={0 0}, upload=516303]: Long tail cancellation due to optimal threshold Dec 30 04:26:43.544528522 (588h57m43.415611557s) [seg_pos={0 0}, upload=516303]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:26:43.544605859 (588h57m43.415688894s) [upload=516303]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:26:43.544610407 (588h57m43.415693452s) [upload=516303]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:26:43.831290249 (588h57m43.702373294s) [upload=516303]: Done uploading segments... Dec 30 04:26:43.831318373 (588h57m43.702401448s) [upload=516303]: Upload finished. err: <nil> Duration: 30.003764587s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.........X..XXX............XX..X.....X..............X.......] duration: 3.519646288s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:27:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.519675063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X............X..............................] duration: 33.993594315s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:26:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 33.993630385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..........X..XXX............XX..X.....X..............X......] duration: 3.606365635s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:26:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.606391244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X............X.............................] duration: 42.813335237s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:26:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 42.813353081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.X..] duration: 6.011960234s trace: https://jaeger.dev.storj.io/trace/7d1963d28d4b4b65 FAIL (2025-12-30 04:25:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 5ARLHWYHA4KKHNFKTGXKK7GNMMJ45POKO5FLXO76Y4UMQU4CYK2Q) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: CBIUOVLFQZTZG3DOH2N7USYTQXMHNL4TVJI5JR2MKBC4JAHIOM2Q) context deadline exceeded ======LOGS====== Dec 30 04:24:10.692551906 (588h55m10.563634951s) [upload=516270]: Starting upload Dec 30 04:24:10.692619645 (588h55m10.56370269s) [upload=516270]: Done starting upload Dec 30 04:24:10.692687665 (588h55m10.5637707s) [upload=516270]: Uploading segments... Dec 30 04:24:10.693300932 (588h55m10.564383967s) [upload=516270]: Got next segment. Inline: false Dec 30 04:24:10.693364874 (588h55m10.564447909s) [upload=516270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:24:10.693370915 (588h55m10.56445395s) [upload=516270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:24:10.936045235 (588h55m10.807128281s) [seg_pos={0 0}, upload=516270]: Begin upload segment... Dec 30 04:24:10.936096443 (588h55m10.807179489s) [seg_pos={0 0}, upload=516270]: Done begin upload segment. Dec 30 04:24:10.936124356 (588h55m10.807207391s) [noise=true , address=45.91.116.5:4300 , piece_id=LKM2QGFSDFACT3TLHVJUIGDPYC3QIIMFBL7X637Y23T277YYOTTA, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.936246569 (588h55m10.807329614s) [noise=true , address=145.79.227.47:4265 , piece_id=O4UAOZFBJAHQ54F5FE3JDYBWPBOYLRNAFBNGZANPHRQDHSFU2DBA, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.936783641 (588h55m10.807866686s) [noise=true , address=45.91.116.12:4225 , piece_id=2ZXN22GTCEG4R7BVSUR3JXKARH43U2OLOEQOJP5IDHTPMIPUF44Q, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.937567564 (588h55m10.8086506s) [noise=true , address=145.79.227.7:4294 , piece_id=KLZFGUCKZ32LCLMIS7233TJ43AJ7YTK3FCRT3NHPEGAFUJ2VLS5Q, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.937878737 (588h55m10.808961772s) [noise=true , address=145.79.227.22:4509 , piece_id=TS64C6T6WVP6R3QLA5VEQRTEBCEW3HI7TMVVQ7D7TJ5E46LQAFKA, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.938079800 (588h55m10.809162835s) [noise=true , address=77.90.31.10:4519 , piece_id=LOPVRWJUBKH5KW3VFYNY6G73N5ACPDRYR3UOE5KZUTMXJS6DU72Q, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.938449884 (588h55m10.809532919s) [noise=true , address=145.79.227.5:4221 , piece_id=6FWVOIJAOVOLAHPX3MZGHY4O2UAR5HOSZVZKTZG4FJU3NRKJFGBA, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.938668090 (588h55m10.809751136s) [noise=true , address=145.79.227.15:4876 , piece_id=CX2Z4COPB5B5MU5BJT3GGWN7R3BBR64ROGDMVBHRO53EFVK7FLPQ, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.939025040 (588h55m10.810108085s) [noise=true , address=45.91.116.14:4192 , piece_id=CBIUOVLFQZTZG3DOH2N7USYTQXMHNL4TVJI5JR2MKBC4JAHIOM2Q, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.939378182 (588h55m10.810461217s) [noise=true , address=145.79.227.31:4798 , piece_id=X44PWTOGMKAIK2UKUVSJEQ4JE2RHJCBBZPE3QUZ7OK3DKBUJ23UQ, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.939633618 (588h55m10.810716673s) [noise=true , address=145.79.227.26:4654 , piece_id=POLK7Y4ZRAVJFGAQE7FPDH6SRXTO5GJFXN2CU4D4LLEVDNFWRSNA, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.940275060 (588h55m10.811358105s) [noise=true , address=77.90.31.16:4735 , piece_id=5ARLHWYHA4KKHNFKTGXKK7GNMMJ45POKO5FLXO76Y4UMQU4CYK2Q, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.940898568 (588h55m10.811981603s) [noise=true , address=145.79.227.50:4373 , piece_id=5YEGNXQUUI62P6SEKXCJ52675NAUJ4OBVOSLBA7GHLX75HRAMH2A, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.942431677 (588h55m10.813514712s) [noise=true , address=145.79.227.13:4802 , piece_id=NHA4KS7EBVJ3FMJJTVN3LTUPAEBJPCUNSGLLY27BFVWXCVYNH54A, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.942954422 (588h55m10.814037467s) [noise=true , address=145.79.227.17:4971 , piece_id=DD6YB7PSND36JGROAYU7LS7FJRSVQ6G2A5Z2WZR3I65SF2SDI5WQ, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:10.943197966 (588h55m10.814281011s) [noise=true , address=145.79.227.6:4258 , piece_id=DRDLV5QAMH6GS3C7OKWHUATIZHXIWFUSZEMBNX3SRG5KUQSBEWLA, seg_pos={0 0}, upload=516270]: Uploading piece... Dec 30 04:24:12.602959113 (588h55m12.474042148s) [upload=516270]: Next returned nil segment Dec 30 04:24:12.602971716 (588h55m12.474054752s) [upload=516270]: Waiting for error group managing segments... Dec 30 04:24:12.917199245 (588h55m12.78828228s) [noise=true , address=145.79.227.50:4373 , piece_id=5YEGNXQUUI62P6SEKXCJ52675NAUJ4OBVOSLBA7GHLX75HRAMH2A, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.922465322 (588h55m12.793548357s) [noise=true , address=145.79.227.22:4509 , piece_id=TS64C6T6WVP6R3QLA5VEQRTEBCEW3HI7TMVVQ7D7TJ5E46LQAFKA, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.922637260 (588h55m12.793720295s) [noise=true , address=145.79.227.7:4294 , piece_id=KLZFGUCKZ32LCLMIS7233TJ43AJ7YTK3FCRT3NHPEGAFUJ2VLS5Q, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.932715001 (588h55m12.803798036s) [noise=true , address=145.79.227.47:4265 , piece_id=O4UAOZFBJAHQ54F5FE3JDYBWPBOYLRNAFBNGZANPHRQDHSFU2DBA, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.933158846 (588h55m12.804241882s) [noise=true , address=145.79.227.13:4802 , piece_id=NHA4KS7EBVJ3FMJJTVN3LTUPAEBJPCUNSGLLY27BFVWXCVYNH54A, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.936552509 (588h55m12.807635554s) [noise=true , address=145.79.227.15:4876 , piece_id=CX2Z4COPB5B5MU5BJT3GGWN7R3BBR64ROGDMVBHRO53EFVK7FLPQ, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.936564161 (588h55m12.807647196s) [noise=true , address=145.79.227.17:4971 , piece_id=DD6YB7PSND36JGROAYU7LS7FJRSVQ6G2A5Z2WZR3I65SF2SDI5WQ, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.943615318 (588h55m12.814698353s) [noise=true , address=145.79.227.26:4654 , piece_id=POLK7Y4ZRAVJFGAQE7FPDH6SRXTO5GJFXN2CU4D4LLEVDNFWRSNA, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.945439142 (588h55m12.816522177s) [noise=true , address=145.79.227.6:4258 , piece_id=DRDLV5QAMH6GS3C7OKWHUATIZHXIWFUSZEMBNX3SRG5KUQSBEWLA, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.946555477 (588h55m12.817638522s) [noise=true , address=145.79.227.5:4221 , piece_id=6FWVOIJAOVOLAHPX3MZGHY4O2UAR5HOSZVZKTZG4FJU3NRKJFGBA, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:12.952769831 (588h55m12.823852866s) [noise=true , address=145.79.227.31:4798 , piece_id=X44PWTOGMKAIK2UKUVSJEQ4JE2RHJCBBZPE3QUZ7OK3DKBUJ23UQ, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:13.118217291 (588h55m12.989300326s) [noise=true , address=77.90.31.16:4735 , piece_id=5ARLHWYHA4KKHNFKTGXKK7GNMMJ45POKO5FLXO76Y4UMQU4CYK2Q, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:13.171621808 (588h55m13.042704843s) [noise=true , address=45.91.116.14:4192 , piece_id=CBIUOVLFQZTZG3DOH2N7USYTQXMHNL4TVJI5JR2MKBC4JAHIOM2Q, seg_pos={0 0}, upload=516270]: Done uploading piece. err: <nil> Dec 30 04:24:13.171645834 (588h55m13.042728869s) [seg_pos={0 0}, upload=516270]: Segment reached optimal threshold of 13 pieces. Dec 30 04:24:13.171843520 (588h55m13.042926555s) [noise=true , address=77.90.31.10:4519 , piece_id=LOPVRWJUBKH5KW3VFYNY6G73N5ACPDRYR3UOE5KZUTMXJS6DU72Q, seg_pos={0 0}, upload=516270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 04:24:13.171849191 (588h55m13.042932226s) [noise=true , address=77.90.31.10:4519 , piece_id=LOPVRWJUBKH5KW3VFYNY6G73N5ACPDRYR3UOE5KZUTMXJS6DU72Q, seg_pos={0 0}, upload=516270]: Long tail cancellation due to optimal threshold Dec 30 04:24:13.172094448 (588h55m13.043177483s) [noise=true , address=45.91.116.5:4300 , piece_id=LKM2QGFSDFACT3TLHVJUIGDPYC3QIIMFBL7X637Y23T277YYOTTA, seg_pos={0 0}, upload=516270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: read tcp 188.166.225.170:50358->45.91.116.5:4300: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:24:13.172100730 (588h55m13.043183765s) [noise=true , address=45.91.116.5:4300 , piece_id=LKM2QGFSDFACT3TLHVJUIGDPYC3QIIMFBL7X637Y23T277YYOTTA, seg_pos={0 0}, upload=516270]: Long tail cancellation due to optimal threshold Dec 30 04:24:13.172122552 (588h55m13.043205597s) [noise=true , address=45.91.116.12:4225 , piece_id=2ZXN22GTCEG4R7BVSUR3JXKARH43U2OLOEQOJP5IDHTPMIPUF44Q, seg_pos={0 0}, upload=516270]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 04:24:13.172141758 (588h55m13.043224793s) [noise=true , address=45.91.116.12:4225 , piece_id=2ZXN22GTCEG4R7BVSUR3JXKARH43U2OLOEQOJP5IDHTPMIPUF44Q, seg_pos={0 0}, upload=516270]: Long tail cancellation due to optimal threshold Dec 30 04:24:13.172271335 (588h55m13.04335439s) [seg_pos={0 0}, upload=516270]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:24:13.172373890 (588h55m13.043456935s) [upload=516270]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:24:13.172381134 (588h55m13.043464169s) [upload=516270]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:24:13.454795761 (588h55m13.325878796s) [upload=516270]: Done uploading segments... Dec 30 04:24:13.454821200 (588h55m13.325904265s) [upload=516270]: Upload finished. err: <nil> Duration: 6.011985603s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...........X..XXX............XX..X.....X..............X.....] duration: 3.447597669s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:25:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447625991s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)