Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X.] duration: 3.522038498s trace: https://jaeger.dev.storj.io/trace/144ba4403f89aa0d FAIL (2025-12-30 11:10:30 UTC) orbiter warning: context deadline exceeded Duration: 3.522065479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X....................X.................X..........X] duration: 1m0.005810166s trace: https://jaeger.dev.storj.io/trace/7e76a273d363be9c FAIL (2025-12-30 11:10:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=145.79.227.30:4772 , piece_id=4KR7T6FABU7U6X7DFE3PT5FY7COW72ERH2YMSZP6HRZDCRBVMUSA, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.588717399 (595h40m11.459800444s) [noise=true , address=77.90.31.18:4834 , piece_id=X2LUGTDOKPINILRQ4LZFEYTIZQXGQZZH7QFOXVETB6ISNV7FJJUA, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.589225266 (595h40m11.460308311s) [noise=true , address=145.79.227.33:4881 , piece_id=IYAJCMXSFIF42HPBXOBHLOXJDISLICXQRN57LA4YWJRS4BJF2R7Q, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.589929326 (595h40m11.461012371s) [noise=true , address=145.79.227.35:4516 , piece_id=W4ZBRRIKLPKIH6OIGRNAAPGVXASY7VRE2DBNYKL4BY4E57FTYJMQ, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.590588291 (595h40m11.461671326s) [noise=true , address=193.149.164.143:4699 , piece_id=IZR56FCFGLJJWQOB7JWXWUVHDMUNBMUSSAVSLP6V5HYWSYQGFXGA, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.591057976 (595h40m11.462141011s) [noise=true , address=145.79.227.34:4930 , piece_id=26WUNUIGYRLTWTE4PS6OQCWZIAYDYJWTZ7NIGQ7TT5DGWUBRI3FQ, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.591457727 (595h40m11.462540772s) [noise=true , address=145.79.227.40:4075 , piece_id=NYZLPHKQ3HAZAEUZAWPETTQZQBHM3WG63ZIO2L35LZ6GCFUCMZJQ, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.591917652 (595h40m11.463000697s) [noise=true , address=64.49.13.169:4711 , piece_id=5VNYED4CHERVBIGQQYPFGXWARUAOAWKLFNCGIJ7Z4NHWQV3RDXOQ, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.592388489 (595h40m11.463471534s) [noise=true , address=145.79.227.15:4888 , piece_id=RRO7KVYBT2EXPLFZOSIJ4DEL66PUFBCWGFT6XTMZUTJHWTLXXXYQ, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.593679017 (595h40m11.464762062s) [noise=true , address=77.90.31.10:4531 , piece_id=TP5R3DSYZLOH2XGYMPYQG4BT6XEJX7OAIHYNQ2QX5XGUNFFXXW5Q, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.596079137 (595h40m11.467162172s) [noise=true , address=45.91.116.5:4317 , piece_id=4GIJ24OYABBDSQZ2YXNDLG4KFTZAWYT4HYXIQEA4RHTFH57ZB6KA, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.596798707 (595h40m11.467881752s) [noise=true , address=45.91.116.10:4105 , piece_id=7ZFYP3ZXVOEM27YF32VI64FB3I5OH5DXPM3ZYJA7TGFZAG6WKQCQ, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.597203397 (595h40m11.468286442s) [noise=true , address=145.79.227.6:4273 , piece_id=GDGOM5YEDPVBMGMC25RY5ZQT2SCI2U5JOVNL2FRF2ZS34TX3Y7OQ, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.597612226 (595h40m11.468695261s) [noise=true , address=145.79.227.29:4966 , piece_id=XVA7FEMHAGTWVFBTV6GFDD2NMRA5X653PTIW2QUCWDD4FCOU3RDA, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.598124432 (595h40m11.469207477s) [noise=true , address=77.90.31.12:4603 , piece_id=4AEVTGF4JKVDVBJNLXZNKR4WQKUUTA4VVETHFGBTSLDNTZOT4SCA, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:11.598657797 (595h40m11.469740832s) [noise=true , address=145.79.227.44:4215 , piece_id=47QQMRXLZHPYFQX7SVECHGFSIP3PA3JL2J2MQ5JI6XO3NSMXR7XA, seg_pos={0 0}, upload=522170]: Uploading piece... Dec 30 11:09:12.978217866 (595h40m12.849300911s) [upload=522170]: Next returned nil segment Dec 30 11:09:12.978240730 (595h40m12.849323775s) [upload=522170]: Waiting for error group managing segments... Dec 30 11:09:13.452321925 (595h40m13.32340499s) [noise=true , address=145.79.227.43:4186 , piece_id=MZBL6P7CL53F4IXH63MTNFQJRB4B4FDCIA3ABDXINZMN6X5L6QTA, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:13.453285149 (595h40m13.324368194s) [noise=true , address=145.79.227.29:4966 , piece_id=XVA7FEMHAGTWVFBTV6GFDD2NMRA5X653PTIW2QUCWDD4FCOU3RDA, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:13.461029075 (595h40m13.33211211s) [noise=true , address=145.79.227.4:4211 , piece_id=WRATJGOB5HBGVJFI4GAWJBRKEXPLGJT7PITWN3IIL3C25HKIRPSQ, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:13.463151366 (595h40m13.334234411s) [noise=true , address=145.79.227.15:4888 , piece_id=RRO7KVYBT2EXPLFZOSIJ4DEL66PUFBCWGFT6XTMZUTJHWTLXXXYQ, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:13.464109249 (595h40m13.335192284s) [noise=true , address=145.79.227.6:4273 , piece_id=GDGOM5YEDPVBMGMC25RY5ZQT2SCI2U5JOVNL2FRF2ZS34TX3Y7OQ, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:13.621185878 (595h40m13.492268923s) [noise=true , address=77.90.31.10:4531 , piece_id=TP5R3DSYZLOH2XGYMPYQG4BT6XEJX7OAIHYNQ2QX5XGUNFFXXW5Q, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:13.706914428 (595h40m13.577997463s) [noise=true , address=77.90.31.18:4834 , piece_id=X2LUGTDOKPINILRQ4LZFEYTIZQXGQZZH7QFOXVETB6ISNV7FJJUA, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:14.150781060 (595h40m14.021864095s) [noise=true , address=45.91.116.5:4317 , piece_id=4GIJ24OYABBDSQZ2YXNDLG4KFTZAWYT4HYXIQEA4RHTFH57ZB6KA, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:14.294569896 (595h40m14.165652941s) [noise=true , address=193.149.164.143:4699 , piece_id=IZR56FCFGLJJWQOB7JWXWUVHDMUNBMUSSAVSLP6V5HYWSYQGFXGA, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:14.357738109 (595h40m14.228821144s) [noise=true , address=145.79.227.33:4881 , piece_id=IYAJCMXSFIF42HPBXOBHLOXJDISLICXQRN57LA4YWJRS4BJF2R7Q, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:14.385421422 (595h40m14.256504457s) [noise=true , address=145.79.227.44:4215 , piece_id=47QQMRXLZHPYFQX7SVECHGFSIP3PA3JL2J2MQ5JI6XO3NSMXR7XA, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:14.396251935 (595h40m14.267334971s) [noise=true , address=145.79.227.30:4772 , piece_id=4KR7T6FABU7U6X7DFE3PT5FY7COW72ERH2YMSZP6HRZDCRBVMUSA, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:14.423516801 (595h40m14.294599836s) [noise=true , address=145.79.227.34:4930 , piece_id=26WUNUIGYRLTWTE4PS6OQCWZIAYDYJWTZ7NIGQ7TT5DGWUBRI3FQ, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:14.489542765 (595h40m14.3606258s) [noise=true , address=145.79.227.35:4516 , piece_id=W4ZBRRIKLPKIH6OIGRNAAPGVXASY7VRE2DBNYKL4BY4E57FTYJMQ, seg_pos={0 0}, upload=522170]: Done uploading piece. err: <nil> Dec 30 11:09:14.489563354 (595h40m14.360646399s) [seg_pos={0 0}, upload=522170]: Segment reached optimal threshold of 14 pieces. Dec 30 11:09:14.489780799 (595h40m14.360863834s) [noise=true , address=64.49.13.169:4711 , piece_id=5VNYED4CHERVBIGQQYPFGXWARUAOAWKLFNCGIJ7Z4NHWQV3RDXOQ, seg_pos={0 0}, upload=522170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:33104->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:33104->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33104->64.49.13.169:4711: use of closed network connection Dec 30 11:09:14.489787311 (595h40m14.360870346s) [noise=true , address=64.49.13.169:4711 , piece_id=5VNYED4CHERVBIGQQYPFGXWARUAOAWKLFNCGIJ7Z4NHWQV3RDXOQ, seg_pos={0 0}, upload=522170]: Long tail cancellation due to optimal threshold Dec 30 11:09:14.489882853 (595h40m14.360965888s) [noise=true , address=145.79.227.40:4075 , piece_id=NYZLPHKQ3HAZAEUZAWPETTQZQBHM3WG63ZIO2L35LZ6GCFUCMZJQ, seg_pos={0 0}, upload=522170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 11:09:14.489922348 (595h40m14.361005383s) [noise=true , address=145.79.227.40:4075 , piece_id=NYZLPHKQ3HAZAEUZAWPETTQZQBHM3WG63ZIO2L35LZ6GCFUCMZJQ, seg_pos={0 0}, upload=522170]: Long tail cancellation due to optimal threshold Dec 30 11:09:14.490082583 (595h40m14.361165628s) [noise=true , address=77.90.31.12:4603 , piece_id=4AEVTGF4JKVDVBJNLXZNKR4WQKUUTA4VVETHFGBTSLDNTZOT4SCA, seg_pos={0 0}, upload=522170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126Sou5pnMm2eE2FJpv6WYX2VSqJvZZf9rFFJD6hFYDetRD28zu): protocol: expected piece hash; context canceled; EOF Dec 30 11:09:14.490090629 (595h40m14.361173664s) [noise=true , address=77.90.31.12:4603 , piece_id=4AEVTGF4JKVDVBJNLXZNKR4WQKUUTA4VVETHFGBTSLDNTZOT4SCA, seg_pos={0 0}, upload=522170]: Long tail cancellation due to optimal threshold Dec 30 11:09:14.490129102 (595h40m14.361212147s) [noise=true , address=45.91.116.10:4105 , piece_id=7ZFYP3ZXVOEM27YF32VI64FB3I5OH5DXPM3ZYJA7TGFZAG6WKQCQ, seg_pos={0 0}, upload=522170]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12D2biCCeFVigmPrTL1DchNPy9gtJZbaZLowMVajZrcTaizvPXV): protocol: expected piece hash; context canceled; EOF Dec 30 11:09:14.490136887 (595h40m14.361219932s) [noise=true , address=45.91.116.10:4105 , piece_id=7ZFYP3ZXVOEM27YF32VI64FB3I5OH5DXPM3ZYJA7TGFZAG6WKQCQ, seg_pos={0 0}, upload=522170]: Long tail cancellation due to optimal threshold Dec 30 11:09:14.490159851 (595h40m14.361242886s) [seg_pos={0 0}, upload=522170]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 11:09:14.490303895 (595h40m14.36138693s) [upload=522170]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:09:14.490310237 (595h40m14.361393272s) [upload=522170]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:09:14.797447955 (595h40m14.66853099s) [upload=522170]: Done uploading segments... Dec 30 11:09:14.797467922 (595h40m14.668550957s) [upload=522170]: Upload finished. err: <nil> Duration: 1m0.005839642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X...X.X..XX.XX........X.......X..XX..X.X.X..........X....X..] duration: 5.931384552s trace: https://jaeger.dev.storj.io/trace/52eb849e66c0ad7f FAIL (2025-12-30 11:09:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: AVX3QPC5INFRUOGFV2BZNWBJAJAFU5YRMM4UHBIXPSACFFZC4GIQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: A4Q2LLRSQVF7YFVH3CO45IVUUA2P5IBUDBGIJ2XYUH5PRW7GNHWA) context deadline exceeded ======LOGS====== Dec 30 11:08:41.930193964 (595h39m41.801277009s) [upload=522163]: Starting upload Dec 30 11:08:41.930294355 (595h39m41.801377401s) [upload=522163]: Done starting upload Dec 30 11:08:41.930503133 (595h39m41.801586178s) [upload=522163]: Uploading segments... Dec 30 11:08:41.930794498 (595h39m41.801877544s) [upload=522163]: Got next segment. Inline: false Dec 30 11:08:41.930865904 (595h39m41.801948949s) [upload=522163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:08:41.930871615 (595h39m41.80195466s) [upload=522163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:08:42.174977090 (595h39m42.046060135s) [seg_pos={0 0}, upload=522163]: Begin upload segment... Dec 30 11:08:42.175091036 (595h39m42.046174091s) [seg_pos={0 0}, upload=522163]: Done begin upload segment. Dec 30 11:08:42.175129199 (595h39m42.046212244s) [noise=true , address=145.79.227.33:4870 , piece_id=HETTKKUDFHLZUEAQFRGFEWZZA4CTY64PTDQIPLTBPHHTXB3ZFIAA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.176409798 (595h39m42.047492833s) [noise=true , address=145.79.227.38:4978 , piece_id=T6WX6YEBEA2SGFMKWJJZNPF56LUCGO7JOWMIWYYNVMQQVR6RSAXA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.176908497 (595h39m42.047991542s) [noise=true , address=145.79.227.16:4912 , piece_id=PP36IYIQWVB46TVRL4INZFRWY6J7JY2JBQ4XRMVQEL6JY6BVKJSQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.177310573 (595h39m42.048393608s) [noise=true , address=145.79.227.4:4185 , piece_id=6OVFEGDHGASU2UFI6S4AMA55RAWYN7FTLQAW5NGPNIQ7RVTKLNVQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.177641362 (595h39m42.048724407s) [noise=true , address=145.79.227.22:4510 , piece_id=I5JHQ3VSWZ6ZX6S4TVHOJYPZ2DSMVDVEA7O7HSAH3EBPPAXFOYIQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.178010245 (595h39m42.04909328s) [noise=true , address=145.79.227.2:4114 , piece_id=NO7ECNOCBV24YT47GABM65CIRI7IONC7LZBWAULM2IO7BLZVXWNQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.178421569 (595h39m42.049504604s) [noise=true , address=45.91.116.7:4011 , piece_id=AVX3QPC5INFRUOGFV2BZNWBJAJAFU5YRMM4UHBIXPSACFFZC4GIQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.178830908 (595h39m42.049913953s) [noise=true , address=145.79.227.17:4971 , piece_id=XFCX3EDLCEOQZJATYJ4YQICFE2K2UTRGJHSL4LQH647XZKBETOIQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.179400092 (595h39m42.050483137s) [noise=true , address=145.79.227.11:4732 , piece_id=X5X2G63ME7TMUOIVTNGUQ53ZZGDQBPXYCSZ4WXQKYAM7EKXG7NCQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.179796887 (595h39m42.050879932s) [noise=true , address=45.91.116.5:4298 , piece_id=HMUMQSAAXJNGLR4JCB4AQLWOVCGK6HGGI7HV6T76NFVVIFEVYDUA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.180688275 (595h39m42.05177132s) [noise=true , address=145.79.227.34:4907 , piece_id=MU5M7KVBYZCMIEWZBAF367PF7Z2FAGDEFATORET2CQPC6HNTJ5PA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.183089998 (595h39m42.054173043s) [noise=true , address=145.79.227.46:4601 , piece_id=YJIS3KJYKCZ6BXKFKL7KMQOSO2ZH77N777LEBD2ZVGLXVKOSYIYA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.185609736 (595h39m42.056692781s) [noise=true , address=145.79.227.7:4294 , piece_id=P7C7E6OHBYHKDABIMZJXDINOKYXLHWYTZR5X6MPGGAGJLCDEU7NA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.187769598 (595h39m42.058852643s) [noise=true , address=45.91.116.10:4084 , piece_id=A4Q2LLRSQVF7YFVH3CO45IVUUA2P5IBUDBGIJ2XYUH5PRW7GNHWA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.189982422 (595h39m42.061065467s) [noise=true , address=145.79.227.15:4874 , piece_id=EZD6XPIHMG5VJSZZ4MNC2ISFR7XZ2B7PESS56P467L3I755EZK3Q, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.192323750 (595h39m42.063406795s) [noise=true , address=145.79.227.24:4581 , piece_id=6RLXFNTBC5XOBSQHYQDEERNJPGASG366MZR7GLI3LC4EEOZBR2VQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:43.876228147 (595h39m43.747311182s) [upload=522163]: Next returned nil segment Dec 30 11:08:43.876248305 (595h39m43.74733135s) [upload=522163]: Waiting for error group managing segments... Dec 30 11:08:44.161317801 (595h39m44.032400836s) [noise=true , address=145.79.227.17:4971 , piece_id=XFCX3EDLCEOQZJATYJ4YQICFE2K2UTRGJHSL4LQH647XZKBETOIQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.168562247 (595h39m44.039645302s) [noise=true , address=145.79.227.22:4510 , piece_id=I5JHQ3VSWZ6ZX6S4TVHOJYPZ2DSMVDVEA7O7HSAH3EBPPAXFOYIQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.170229701 (595h39m44.041312736s) [noise=true , address=145.79.227.4:4185 , piece_id=6OVFEGDHGASU2UFI6S4AMA55RAWYN7FTLQAW5NGPNIQ7RVTKLNVQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.179381918 (595h39m44.050464964s) [noise=true , address=145.79.227.16:4912 , piece_id=PP36IYIQWVB46TVRL4INZFRWY6J7JY2JBQ4XRMVQEL6JY6BVKJSQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.196284503 (595h39m44.067367538s) [noise=true , address=145.79.227.38:4978 , piece_id=T6WX6YEBEA2SGFMKWJJZNPF56LUCGO7JOWMIWYYNVMQQVR6RSAXA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.197132509 (595h39m44.068215554s) [noise=true , address=145.79.227.7:4294 , piece_id=P7C7E6OHBYHKDABIMZJXDINOKYXLHWYTZR5X6MPGGAGJLCDEU7NA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.200879653 (595h39m44.071962698s) [noise=true , address=145.79.227.15:4874 , piece_id=EZD6XPIHMG5VJSZZ4MNC2ISFR7XZ2B7PESS56P467L3I755EZK3Q, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.248883551 (595h39m44.119966586s) [noise=true , address=145.79.227.46:4601 , piece_id=YJIS3KJYKCZ6BXKFKL7KMQOSO2ZH77N777LEBD2ZVGLXVKOSYIYA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.253638996 (595h39m44.124722031s) [noise=true , address=145.79.227.11:4732 , piece_id=X5X2G63ME7TMUOIVTNGUQ53ZZGDQBPXYCSZ4WXQKYAM7EKXG7NCQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.267419336 (595h39m44.138502381s) [noise=true , address=145.79.227.24:4581 , piece_id=6RLXFNTBC5XOBSQHYQDEERNJPGASG366MZR7GLI3LC4EEOZBR2VQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.337523826 (595h39m44.208606861s) [noise=true , address=145.79.227.2:4114 , piece_id=NO7ECNOCBV24YT47GABM65CIRI7IONC7LZBWAULM2IO7BLZVXWNQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.505258448 (595h39m44.376341483s) [noise=true , address=45.91.116.7:4011 , piece_id=AVX3QPC5INFRUOGFV2BZNWBJAJAFU5YRMM4UHBIXPSACFFZC4GIQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.523431563 (595h39m44.394514608s) [noise=true , address=45.91.116.10:4084 , piece_id=A4Q2LLRSQVF7YFVH3CO45IVUUA2P5IBUDBGIJ2XYUH5PRW7GNHWA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.523459386 (595h39m44.394542431s) [seg_pos={0 0}, upload=522163]: Segment reached optimal threshold of 13 pieces. Dec 30 11:08:44.523773104 (595h39m44.394856159s) [noise=true , address=145.79.227.34:4907 , piece_id=MU5M7KVBYZCMIEWZBAF367PF7Z2FAGDEFATORET2CQPC6HNTJ5PA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 11:08:44.523782932 (595h39m44.394865977s) [noise=true , address=145.79.227.34:4907 , piece_id=MU5M7KVBYZCMIEWZBAF367PF7Z2FAGDEFATORET2CQPC6HNTJ5PA, seg_pos={0 0}, upload=522163]: Long tail cancellation due to optimal threshold Dec 30 11:08:44.523919292 (595h39m44.395002337s) [noise=true , address=45.91.116.5:4298 , piece_id=HMUMQSAAXJNGLR4JCB4AQLWOVCGK6HGGI7HV6T76NFVVIFEVYDUA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 11:08:44.523962404 (595h39m44.395045449s) [noise=true , address=45.91.116.5:4298 , piece_id=HMUMQSAAXJNGLR4JCB4AQLWOVCGK6HGGI7HV6T76NFVVIFEVYDUA, seg_pos={0 0}, upload=522163]: Long tail cancellation due to optimal threshold Dec 30 11:08:44.524154500 (595h39m44.395237555s) [noise=true , address=145.79.227.33:4870 , piece_id=HETTKKUDFHLZUEAQFRGFEWZZA4CTY64PTDQIPLTBPHHTXB3ZFIAA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 11:08:44.524166332 (595h39m44.395249387s) [noise=true , address=145.79.227.33:4870 , piece_id=HETTKKUDFHLZUEAQFRGFEWZZA4CTY64PTDQIPLTBPHHTXB3ZFIAA, seg_pos={0 0}, upload=522163]: Long tail cancellation due to optimal threshold Dec 30 11:08:44.524201709 (595h39m44.395284754s) [seg_pos={0 0}, upload=522163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:08:44.524304986 (595h39m44.395388031s) [upload=522163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:08:44.524311809 (595h39m44.395394844s) [upload=522163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:08:44.808621389 (595h39m44.679704444s) [upload=522163]: Done uploading segments... Dec 30 11:08:44.808647359 (595h39m44.679730404s) [upload=522163]: Upload finished. err: <nil> Duration: 5.931401153s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.....X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.X] duration: 5.001013448s trace: https://jaeger.dev.storj.io/trace/144ba4403f89aa0d FAIL (2025-12-30 11:09:33 UTC) orbiter warning: context deadline exceeded Duration: 5.001039327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX...X.X..XX.XX........X.......X..XX..X.X.X..........X....X.] duration: 8.40301049s trace: https://jaeger.dev.storj.io/trace/52eb849e66c0ad7f FAIL (2025-12-30 11:09:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: AVX3QPC5INFRUOGFV2BZNWBJAJAFU5YRMM4UHBIXPSACFFZC4GIQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: A4Q2LLRSQVF7YFVH3CO45IVUUA2P5IBUDBGIJ2XYUH5PRW7GNHWA) context deadline exceeded ======LOGS====== Dec 30 11:08:41.930193964 (595h39m41.801277009s) [upload=522163]: Starting upload Dec 30 11:08:41.930294355 (595h39m41.801377401s) [upload=522163]: Done starting upload Dec 30 11:08:41.930503133 (595h39m41.801586178s) [upload=522163]: Uploading segments... Dec 30 11:08:41.930794498 (595h39m41.801877544s) [upload=522163]: Got next segment. Inline: false Dec 30 11:08:41.930865904 (595h39m41.801948949s) [upload=522163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:08:41.930871615 (595h39m41.80195466s) [upload=522163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:08:42.174977090 (595h39m42.046060135s) [seg_pos={0 0}, upload=522163]: Begin upload segment... Dec 30 11:08:42.175091036 (595h39m42.046174091s) [seg_pos={0 0}, upload=522163]: Done begin upload segment. Dec 30 11:08:42.175129199 (595h39m42.046212244s) [noise=true , address=145.79.227.33:4870 , piece_id=HETTKKUDFHLZUEAQFRGFEWZZA4CTY64PTDQIPLTBPHHTXB3ZFIAA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.176409798 (595h39m42.047492833s) [noise=true , address=145.79.227.38:4978 , piece_id=T6WX6YEBEA2SGFMKWJJZNPF56LUCGO7JOWMIWYYNVMQQVR6RSAXA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.176908497 (595h39m42.047991542s) [noise=true , address=145.79.227.16:4912 , piece_id=PP36IYIQWVB46TVRL4INZFRWY6J7JY2JBQ4XRMVQEL6JY6BVKJSQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.177310573 (595h39m42.048393608s) [noise=true , address=145.79.227.4:4185 , piece_id=6OVFEGDHGASU2UFI6S4AMA55RAWYN7FTLQAW5NGPNIQ7RVTKLNVQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.177641362 (595h39m42.048724407s) [noise=true , address=145.79.227.22:4510 , piece_id=I5JHQ3VSWZ6ZX6S4TVHOJYPZ2DSMVDVEA7O7HSAH3EBPPAXFOYIQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.178010245 (595h39m42.04909328s) [noise=true , address=145.79.227.2:4114 , piece_id=NO7ECNOCBV24YT47GABM65CIRI7IONC7LZBWAULM2IO7BLZVXWNQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.178421569 (595h39m42.049504604s) [noise=true , address=45.91.116.7:4011 , piece_id=AVX3QPC5INFRUOGFV2BZNWBJAJAFU5YRMM4UHBIXPSACFFZC4GIQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.178830908 (595h39m42.049913953s) [noise=true , address=145.79.227.17:4971 , piece_id=XFCX3EDLCEOQZJATYJ4YQICFE2K2UTRGJHSL4LQH647XZKBETOIQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.179400092 (595h39m42.050483137s) [noise=true , address=145.79.227.11:4732 , piece_id=X5X2G63ME7TMUOIVTNGUQ53ZZGDQBPXYCSZ4WXQKYAM7EKXG7NCQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.179796887 (595h39m42.050879932s) [noise=true , address=45.91.116.5:4298 , piece_id=HMUMQSAAXJNGLR4JCB4AQLWOVCGK6HGGI7HV6T76NFVVIFEVYDUA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.180688275 (595h39m42.05177132s) [noise=true , address=145.79.227.34:4907 , piece_id=MU5M7KVBYZCMIEWZBAF367PF7Z2FAGDEFATORET2CQPC6HNTJ5PA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.183089998 (595h39m42.054173043s) [noise=true , address=145.79.227.46:4601 , piece_id=YJIS3KJYKCZ6BXKFKL7KMQOSO2ZH77N777LEBD2ZVGLXVKOSYIYA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.185609736 (595h39m42.056692781s) [noise=true , address=145.79.227.7:4294 , piece_id=P7C7E6OHBYHKDABIMZJXDINOKYXLHWYTZR5X6MPGGAGJLCDEU7NA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.187769598 (595h39m42.058852643s) [noise=true , address=45.91.116.10:4084 , piece_id=A4Q2LLRSQVF7YFVH3CO45IVUUA2P5IBUDBGIJ2XYUH5PRW7GNHWA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.189982422 (595h39m42.061065467s) [noise=true , address=145.79.227.15:4874 , piece_id=EZD6XPIHMG5VJSZZ4MNC2ISFR7XZ2B7PESS56P467L3I755EZK3Q, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.192323750 (595h39m42.063406795s) [noise=true , address=145.79.227.24:4581 , piece_id=6RLXFNTBC5XOBSQHYQDEERNJPGASG366MZR7GLI3LC4EEOZBR2VQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:43.876228147 (595h39m43.747311182s) [upload=522163]: Next returned nil segment Dec 30 11:08:43.876248305 (595h39m43.74733135s) [upload=522163]: Waiting for error group managing segments... Dec 30 11:08:44.161317801 (595h39m44.032400836s) [noise=true , address=145.79.227.17:4971 , piece_id=XFCX3EDLCEOQZJATYJ4YQICFE2K2UTRGJHSL4LQH647XZKBETOIQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.168562247 (595h39m44.039645302s) [noise=true , address=145.79.227.22:4510 , piece_id=I5JHQ3VSWZ6ZX6S4TVHOJYPZ2DSMVDVEA7O7HSAH3EBPPAXFOYIQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.170229701 (595h39m44.041312736s) [noise=true , address=145.79.227.4:4185 , piece_id=6OVFEGDHGASU2UFI6S4AMA55RAWYN7FTLQAW5NGPNIQ7RVTKLNVQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.179381918 (595h39m44.050464964s) [noise=true , address=145.79.227.16:4912 , piece_id=PP36IYIQWVB46TVRL4INZFRWY6J7JY2JBQ4XRMVQEL6JY6BVKJSQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.196284503 (595h39m44.067367538s) [noise=true , address=145.79.227.38:4978 , piece_id=T6WX6YEBEA2SGFMKWJJZNPF56LUCGO7JOWMIWYYNVMQQVR6RSAXA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.197132509 (595h39m44.068215554s) [noise=true , address=145.79.227.7:4294 , piece_id=P7C7E6OHBYHKDABIMZJXDINOKYXLHWYTZR5X6MPGGAGJLCDEU7NA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.200879653 (595h39m44.071962698s) [noise=true , address=145.79.227.15:4874 , piece_id=EZD6XPIHMG5VJSZZ4MNC2ISFR7XZ2B7PESS56P467L3I755EZK3Q, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.248883551 (595h39m44.119966586s) [noise=true , address=145.79.227.46:4601 , piece_id=YJIS3KJYKCZ6BXKFKL7KMQOSO2ZH77N777LEBD2ZVGLXVKOSYIYA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.253638996 (595h39m44.124722031s) [noise=true , address=145.79.227.11:4732 , piece_id=X5X2G63ME7TMUOIVTNGUQ53ZZGDQBPXYCSZ4WXQKYAM7EKXG7NCQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.267419336 (595h39m44.138502381s) [noise=true , address=145.79.227.24:4581 , piece_id=6RLXFNTBC5XOBSQHYQDEERNJPGASG366MZR7GLI3LC4EEOZBR2VQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.337523826 (595h39m44.208606861s) [noise=true , address=145.79.227.2:4114 , piece_id=NO7ECNOCBV24YT47GABM65CIRI7IONC7LZBWAULM2IO7BLZVXWNQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.505258448 (595h39m44.376341483s) [noise=true , address=45.91.116.7:4011 , piece_id=AVX3QPC5INFRUOGFV2BZNWBJAJAFU5YRMM4UHBIXPSACFFZC4GIQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.523431563 (595h39m44.394514608s) [noise=true , address=45.91.116.10:4084 , piece_id=A4Q2LLRSQVF7YFVH3CO45IVUUA2P5IBUDBGIJ2XYUH5PRW7GNHWA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.523459386 (595h39m44.394542431s) [seg_pos={0 0}, upload=522163]: Segment reached optimal threshold of 13 pieces. Dec 30 11:08:44.523773104 (595h39m44.394856159s) [noise=true , address=145.79.227.34:4907 , piece_id=MU5M7KVBYZCMIEWZBAF367PF7Z2FAGDEFATORET2CQPC6HNTJ5PA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 11:08:44.523782932 (595h39m44.394865977s) [noise=true , address=145.79.227.34:4907 , piece_id=MU5M7KVBYZCMIEWZBAF367PF7Z2FAGDEFATORET2CQPC6HNTJ5PA, seg_pos={0 0}, upload=522163]: Long tail cancellation due to optimal threshold Dec 30 11:08:44.523919292 (595h39m44.395002337s) [noise=true , address=45.91.116.5:4298 , piece_id=HMUMQSAAXJNGLR4JCB4AQLWOVCGK6HGGI7HV6T76NFVVIFEVYDUA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 11:08:44.523962404 (595h39m44.395045449s) [noise=true , address=45.91.116.5:4298 , piece_id=HMUMQSAAXJNGLR4JCB4AQLWOVCGK6HGGI7HV6T76NFVVIFEVYDUA, seg_pos={0 0}, upload=522163]: Long tail cancellation due to optimal threshold Dec 30 11:08:44.524154500 (595h39m44.395237555s) [noise=true , address=145.79.227.33:4870 , piece_id=HETTKKUDFHLZUEAQFRGFEWZZA4CTY64PTDQIPLTBPHHTXB3ZFIAA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 11:08:44.524166332 (595h39m44.395249387s) [noise=true , address=145.79.227.33:4870 , piece_id=HETTKKUDFHLZUEAQFRGFEWZZA4CTY64PTDQIPLTBPHHTXB3ZFIAA, seg_pos={0 0}, upload=522163]: Long tail cancellation due to optimal threshold Dec 30 11:08:44.524201709 (595h39m44.395284754s) [seg_pos={0 0}, upload=522163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:08:44.524304986 (595h39m44.395388031s) [upload=522163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:08:44.524311809 (595h39m44.395394844s) [upload=522163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:08:44.808621389 (595h39m44.679704444s) [upload=522163]: Done uploading segments... Dec 30 11:08:44.808647359 (595h39m44.679730404s) [upload=522163]: Upload finished. err: <nil> Duration: 8.403041029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX...X.X..XX.XX........X.......X..XX..X.X.X..........X....X] duration: 30.004408044s trace: https://jaeger.dev.storj.io/trace/52eb849e66c0ad7f FAIL (2025-12-30 11:09:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: AVX3QPC5INFRUOGFV2BZNWBJAJAFU5YRMM4UHBIXPSACFFZC4GIQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: A4Q2LLRSQVF7YFVH3CO45IVUUA2P5IBUDBGIJ2XYUH5PRW7GNHWA) context deadline exceeded ======LOGS====== Dec 30 11:08:41.930193964 (595h39m41.801277009s) [upload=522163]: Starting upload Dec 30 11:08:41.930294355 (595h39m41.801377401s) [upload=522163]: Done starting upload Dec 30 11:08:41.930503133 (595h39m41.801586178s) [upload=522163]: Uploading segments... Dec 30 11:08:41.930794498 (595h39m41.801877544s) [upload=522163]: Got next segment. Inline: false Dec 30 11:08:41.930865904 (595h39m41.801948949s) [upload=522163]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:08:41.930871615 (595h39m41.80195466s) [upload=522163]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:08:42.174977090 (595h39m42.046060135s) [seg_pos={0 0}, upload=522163]: Begin upload segment... Dec 30 11:08:42.175091036 (595h39m42.046174091s) [seg_pos={0 0}, upload=522163]: Done begin upload segment. Dec 30 11:08:42.175129199 (595h39m42.046212244s) [noise=true , address=145.79.227.33:4870 , piece_id=HETTKKUDFHLZUEAQFRGFEWZZA4CTY64PTDQIPLTBPHHTXB3ZFIAA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.176409798 (595h39m42.047492833s) [noise=true , address=145.79.227.38:4978 , piece_id=T6WX6YEBEA2SGFMKWJJZNPF56LUCGO7JOWMIWYYNVMQQVR6RSAXA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.176908497 (595h39m42.047991542s) [noise=true , address=145.79.227.16:4912 , piece_id=PP36IYIQWVB46TVRL4INZFRWY6J7JY2JBQ4XRMVQEL6JY6BVKJSQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.177310573 (595h39m42.048393608s) [noise=true , address=145.79.227.4:4185 , piece_id=6OVFEGDHGASU2UFI6S4AMA55RAWYN7FTLQAW5NGPNIQ7RVTKLNVQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.177641362 (595h39m42.048724407s) [noise=true , address=145.79.227.22:4510 , piece_id=I5JHQ3VSWZ6ZX6S4TVHOJYPZ2DSMVDVEA7O7HSAH3EBPPAXFOYIQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.178010245 (595h39m42.04909328s) [noise=true , address=145.79.227.2:4114 , piece_id=NO7ECNOCBV24YT47GABM65CIRI7IONC7LZBWAULM2IO7BLZVXWNQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.178421569 (595h39m42.049504604s) [noise=true , address=45.91.116.7:4011 , piece_id=AVX3QPC5INFRUOGFV2BZNWBJAJAFU5YRMM4UHBIXPSACFFZC4GIQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.178830908 (595h39m42.049913953s) [noise=true , address=145.79.227.17:4971 , piece_id=XFCX3EDLCEOQZJATYJ4YQICFE2K2UTRGJHSL4LQH647XZKBETOIQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.179400092 (595h39m42.050483137s) [noise=true , address=145.79.227.11:4732 , piece_id=X5X2G63ME7TMUOIVTNGUQ53ZZGDQBPXYCSZ4WXQKYAM7EKXG7NCQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.179796887 (595h39m42.050879932s) [noise=true , address=45.91.116.5:4298 , piece_id=HMUMQSAAXJNGLR4JCB4AQLWOVCGK6HGGI7HV6T76NFVVIFEVYDUA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.180688275 (595h39m42.05177132s) [noise=true , address=145.79.227.34:4907 , piece_id=MU5M7KVBYZCMIEWZBAF367PF7Z2FAGDEFATORET2CQPC6HNTJ5PA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.183089998 (595h39m42.054173043s) [noise=true , address=145.79.227.46:4601 , piece_id=YJIS3KJYKCZ6BXKFKL7KMQOSO2ZH77N777LEBD2ZVGLXVKOSYIYA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.185609736 (595h39m42.056692781s) [noise=true , address=145.79.227.7:4294 , piece_id=P7C7E6OHBYHKDABIMZJXDINOKYXLHWYTZR5X6MPGGAGJLCDEU7NA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.187769598 (595h39m42.058852643s) [noise=true , address=45.91.116.10:4084 , piece_id=A4Q2LLRSQVF7YFVH3CO45IVUUA2P5IBUDBGIJ2XYUH5PRW7GNHWA, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.189982422 (595h39m42.061065467s) [noise=true , address=145.79.227.15:4874 , piece_id=EZD6XPIHMG5VJSZZ4MNC2ISFR7XZ2B7PESS56P467L3I755EZK3Q, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:42.192323750 (595h39m42.063406795s) [noise=true , address=145.79.227.24:4581 , piece_id=6RLXFNTBC5XOBSQHYQDEERNJPGASG366MZR7GLI3LC4EEOZBR2VQ, seg_pos={0 0}, upload=522163]: Uploading piece... Dec 30 11:08:43.876228147 (595h39m43.747311182s) [upload=522163]: Next returned nil segment Dec 30 11:08:43.876248305 (595h39m43.74733135s) [upload=522163]: Waiting for error group managing segments... Dec 30 11:08:44.161317801 (595h39m44.032400836s) [noise=true , address=145.79.227.17:4971 , piece_id=XFCX3EDLCEOQZJATYJ4YQICFE2K2UTRGJHSL4LQH647XZKBETOIQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.168562247 (595h39m44.039645302s) [noise=true , address=145.79.227.22:4510 , piece_id=I5JHQ3VSWZ6ZX6S4TVHOJYPZ2DSMVDVEA7O7HSAH3EBPPAXFOYIQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.170229701 (595h39m44.041312736s) [noise=true , address=145.79.227.4:4185 , piece_id=6OVFEGDHGASU2UFI6S4AMA55RAWYN7FTLQAW5NGPNIQ7RVTKLNVQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.179381918 (595h39m44.050464964s) [noise=true , address=145.79.227.16:4912 , piece_id=PP36IYIQWVB46TVRL4INZFRWY6J7JY2JBQ4XRMVQEL6JY6BVKJSQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.196284503 (595h39m44.067367538s) [noise=true , address=145.79.227.38:4978 , piece_id=T6WX6YEBEA2SGFMKWJJZNPF56LUCGO7JOWMIWYYNVMQQVR6RSAXA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.197132509 (595h39m44.068215554s) [noise=true , address=145.79.227.7:4294 , piece_id=P7C7E6OHBYHKDABIMZJXDINOKYXLHWYTZR5X6MPGGAGJLCDEU7NA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.200879653 (595h39m44.071962698s) [noise=true , address=145.79.227.15:4874 , piece_id=EZD6XPIHMG5VJSZZ4MNC2ISFR7XZ2B7PESS56P467L3I755EZK3Q, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.248883551 (595h39m44.119966586s) [noise=true , address=145.79.227.46:4601 , piece_id=YJIS3KJYKCZ6BXKFKL7KMQOSO2ZH77N777LEBD2ZVGLXVKOSYIYA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.253638996 (595h39m44.124722031s) [noise=true , address=145.79.227.11:4732 , piece_id=X5X2G63ME7TMUOIVTNGUQ53ZZGDQBPXYCSZ4WXQKYAM7EKXG7NCQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.267419336 (595h39m44.138502381s) [noise=true , address=145.79.227.24:4581 , piece_id=6RLXFNTBC5XOBSQHYQDEERNJPGASG366MZR7GLI3LC4EEOZBR2VQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.337523826 (595h39m44.208606861s) [noise=true , address=145.79.227.2:4114 , piece_id=NO7ECNOCBV24YT47GABM65CIRI7IONC7LZBWAULM2IO7BLZVXWNQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.505258448 (595h39m44.376341483s) [noise=true , address=45.91.116.7:4011 , piece_id=AVX3QPC5INFRUOGFV2BZNWBJAJAFU5YRMM4UHBIXPSACFFZC4GIQ, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.523431563 (595h39m44.394514608s) [noise=true , address=45.91.116.10:4084 , piece_id=A4Q2LLRSQVF7YFVH3CO45IVUUA2P5IBUDBGIJ2XYUH5PRW7GNHWA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: <nil> Dec 30 11:08:44.523459386 (595h39m44.394542431s) [seg_pos={0 0}, upload=522163]: Segment reached optimal threshold of 13 pieces. Dec 30 11:08:44.523773104 (595h39m44.394856159s) [noise=true , address=145.79.227.34:4907 , piece_id=MU5M7KVBYZCMIEWZBAF367PF7Z2FAGDEFATORET2CQPC6HNTJ5PA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 30 11:08:44.523782932 (595h39m44.394865977s) [noise=true , address=145.79.227.34:4907 , piece_id=MU5M7KVBYZCMIEWZBAF367PF7Z2FAGDEFATORET2CQPC6HNTJ5PA, seg_pos={0 0}, upload=522163]: Long tail cancellation due to optimal threshold Dec 30 11:08:44.523919292 (595h39m44.395002337s) [noise=true , address=45.91.116.5:4298 , piece_id=HMUMQSAAXJNGLR4JCB4AQLWOVCGK6HGGI7HV6T76NFVVIFEVYDUA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 11:08:44.523962404 (595h39m44.395045449s) [noise=true , address=45.91.116.5:4298 , piece_id=HMUMQSAAXJNGLR4JCB4AQLWOVCGK6HGGI7HV6T76NFVVIFEVYDUA, seg_pos={0 0}, upload=522163]: Long tail cancellation due to optimal threshold Dec 30 11:08:44.524154500 (595h39m44.395237555s) [noise=true , address=145.79.227.33:4870 , piece_id=HETTKKUDFHLZUEAQFRGFEWZZA4CTY64PTDQIPLTBPHHTXB3ZFIAA, seg_pos={0 0}, upload=522163]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 11:08:44.524166332 (595h39m44.395249387s) [noise=true , address=145.79.227.33:4870 , piece_id=HETTKKUDFHLZUEAQFRGFEWZZA4CTY64PTDQIPLTBPHHTXB3ZFIAA, seg_pos={0 0}, upload=522163]: Long tail cancellation due to optimal threshold Dec 30 11:08:44.524201709 (595h39m44.395284754s) [seg_pos={0 0}, upload=522163]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:08:44.524304986 (595h39m44.395388031s) [upload=522163]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:08:44.524311809 (595h39m44.395394844s) [upload=522163]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:08:44.808621389 (595h39m44.679704444s) [upload=522163]: Done uploading segments... Dec 30 11:08:44.808647359 (595h39m44.679730404s) [upload=522163]: Upload finished. err: <nil> Duration: 30.004430025s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [......X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X.] duration: 3.758600581s trace: https://jaeger.dev.storj.io/trace/7dab9051e9fbf3ad FAIL (2025-12-30 11:08:50 UTC) orbiter warning: context deadline exceeded Duration: 3.758668259s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.........X....................X.................X..........] duration: 33.024920706s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:08:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 33.024942287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.........X....................X.................X.........] duration: 43.634423348s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:08:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 43.634453546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XX...X.X..XX.XX........X.......X..XX..X.X.X..........X....] duration: 5.948899171s trace: https://jaeger.dev.storj.io/trace/1efd62034a49d3b FAIL (2025-12-30 11:08:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ) context deadline exceeded ======LOGS====== Dec 30 11:05:31.076907427 (595h36m30.947990462s) [upload=522119]: Starting upload Dec 30 11:05:31.076967993 (595h36m30.948051038s) [upload=522119]: Done starting upload Dec 30 11:05:31.077053936 (595h36m30.948136971s) [upload=522119]: Uploading segments... Dec 30 11:05:31.077197178 (595h36m30.948280213s) [upload=522119]: Got next segment. Inline: false Dec 30 11:05:31.077233808 (595h36m30.948316843s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:05:31.077238317 (595h36m30.948321352s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:05:31.321780635 (595h36m31.19286368s) [seg_pos={0 0}, upload=522119]: Begin upload segment... Dec 30 11:05:31.321866699 (595h36m31.192949744s) [seg_pos={0 0}, upload=522119]: Done begin upload segment. Dec 30 11:05:31.321897207 (595h36m31.192980242s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.322782493 (595h36m31.193865538s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323184989 (595h36m31.194268024s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323494158 (595h36m31.194577193s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323955717 (595h36m31.195038792s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324400043 (595h36m31.195483088s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324830674 (595h36m31.195913719s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325147958 (595h36m31.196230993s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325505809 (595h36m31.196588844s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.326834970 (595h36m31.197918005s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.328334906 (595h36m31.199417941s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.329770349 (595h36m31.200853394s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.331418517 (595h36m31.202501562s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.332734062 (595h36m31.203817097s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.333628375 (595h36m31.20471142s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.334856594 (595h36m31.205939639s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:33.040067914 (595h36m32.911150959s) [upload=522119]: Next returned nil segment Dec 30 11:05:33.040080628 (595h36m32.911163673s) [upload=522119]: Waiting for error group managing segments... Dec 30 11:05:33.318845022 (595h36m33.189928057s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.324469852 (595h36m33.195552887s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.334622755 (595h36m33.20570579s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.335109842 (595h36m33.206192887s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337738137 (595h36m33.208821182s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337942486 (595h36m33.209025531s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.343300238 (595h36m33.214383283s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.408823275 (595h36m33.27990632s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.600573588 (595h36m33.471656633s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.639243532 (595h36m33.510326567s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.646759033 (595h36m33.517842078s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.678953487 (595h36m33.550036532s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684164679 (595h36m33.555247714s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684188575 (595h36m33.55527161s) [seg_pos={0 0}, upload=522119]: Segment reached optimal threshold of 13 pieces. Dec 30 11:05:33.684483066 (595h36m33.555566101s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684491702 (595h36m33.555574737s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684811281 (595h36m33.555894316s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684819656 (595h36m33.555902701s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684956717 (595h36m33.556039762s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684963410 (595h36m33.556046445s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684981445 (595h36m33.55606448s) [seg_pos={0 0}, upload=522119]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:05:33.685058802 (595h36m33.556141847s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:05:33.685064963 (595h36m33.556147998s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:05:33.979160510 (595h36m33.850243545s) [upload=522119]: Done uploading segments... Dec 30 11:05:33.979184876 (595h36m33.850267921s) [upload=522119]: Upload finished. err: <nil> Duration: 5.948927706s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.......X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.X] duration: 5.001427509s trace: https://jaeger.dev.storj.io/trace/7dab9051e9fbf3ad FAIL (2025-12-30 11:07:50 UTC) orbiter warning: context deadline exceeded Duration: 5.001450353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XX...X.X..XX.XX........X.......X..XX..X.X.X..........X...] duration: 5.786913875s trace: https://jaeger.dev.storj.io/trace/1efd62034a49d3b FAIL (2025-12-30 11:07:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ) context deadline exceeded ======LOGS====== Dec 30 11:05:31.076907427 (595h36m30.947990462s) [upload=522119]: Starting upload Dec 30 11:05:31.076967993 (595h36m30.948051038s) [upload=522119]: Done starting upload Dec 30 11:05:31.077053936 (595h36m30.948136971s) [upload=522119]: Uploading segments... Dec 30 11:05:31.077197178 (595h36m30.948280213s) [upload=522119]: Got next segment. Inline: false Dec 30 11:05:31.077233808 (595h36m30.948316843s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:05:31.077238317 (595h36m30.948321352s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:05:31.321780635 (595h36m31.19286368s) [seg_pos={0 0}, upload=522119]: Begin upload segment... Dec 30 11:05:31.321866699 (595h36m31.192949744s) [seg_pos={0 0}, upload=522119]: Done begin upload segment. Dec 30 11:05:31.321897207 (595h36m31.192980242s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.322782493 (595h36m31.193865538s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323184989 (595h36m31.194268024s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323494158 (595h36m31.194577193s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323955717 (595h36m31.195038792s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324400043 (595h36m31.195483088s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324830674 (595h36m31.195913719s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325147958 (595h36m31.196230993s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325505809 (595h36m31.196588844s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.326834970 (595h36m31.197918005s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.328334906 (595h36m31.199417941s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.329770349 (595h36m31.200853394s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.331418517 (595h36m31.202501562s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.332734062 (595h36m31.203817097s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.333628375 (595h36m31.20471142s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.334856594 (595h36m31.205939639s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:33.040067914 (595h36m32.911150959s) [upload=522119]: Next returned nil segment Dec 30 11:05:33.040080628 (595h36m32.911163673s) [upload=522119]: Waiting for error group managing segments... Dec 30 11:05:33.318845022 (595h36m33.189928057s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.324469852 (595h36m33.195552887s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.334622755 (595h36m33.20570579s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.335109842 (595h36m33.206192887s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337738137 (595h36m33.208821182s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337942486 (595h36m33.209025531s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.343300238 (595h36m33.214383283s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.408823275 (595h36m33.27990632s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.600573588 (595h36m33.471656633s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.639243532 (595h36m33.510326567s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.646759033 (595h36m33.517842078s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.678953487 (595h36m33.550036532s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684164679 (595h36m33.555247714s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684188575 (595h36m33.55527161s) [seg_pos={0 0}, upload=522119]: Segment reached optimal threshold of 13 pieces. Dec 30 11:05:33.684483066 (595h36m33.555566101s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684491702 (595h36m33.555574737s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684811281 (595h36m33.555894316s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684819656 (595h36m33.555902701s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684956717 (595h36m33.556039762s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684963410 (595h36m33.556046445s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684981445 (595h36m33.55606448s) [seg_pos={0 0}, upload=522119]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:05:33.685058802 (595h36m33.556141847s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:05:33.685064963 (595h36m33.556147998s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:05:33.979160510 (595h36m33.850243545s) [upload=522119]: Done uploading segments... Dec 30 11:05:33.979184876 (595h36m33.850267921s) [upload=522119]: Upload finished. err: <nil> Duration: 5.786941136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.........X....................X.................X........] duration: 44.895577909s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:07:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 44.895603698s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [........X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X.] duration: 3.444621931s trace: https://jaeger.dev.storj.io/trace/1e8ce1f3be3ecd1 FAIL (2025-12-30 11:07:04 UTC) orbiter warning: context deadline exceeded Duration: 3.444652278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....XX...X.X..XX.XX........X.......X..XX..X.X.X..........X..] duration: 6.016598724s trace: https://jaeger.dev.storj.io/trace/1efd62034a49d3b FAIL (2025-12-30 11:06:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ) context deadline exceeded ======LOGS====== Dec 30 11:05:31.076907427 (595h36m30.947990462s) [upload=522119]: Starting upload Dec 30 11:05:31.076967993 (595h36m30.948051038s) [upload=522119]: Done starting upload Dec 30 11:05:31.077053936 (595h36m30.948136971s) [upload=522119]: Uploading segments... Dec 30 11:05:31.077197178 (595h36m30.948280213s) [upload=522119]: Got next segment. Inline: false Dec 30 11:05:31.077233808 (595h36m30.948316843s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:05:31.077238317 (595h36m30.948321352s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:05:31.321780635 (595h36m31.19286368s) [seg_pos={0 0}, upload=522119]: Begin upload segment... Dec 30 11:05:31.321866699 (595h36m31.192949744s) [seg_pos={0 0}, upload=522119]: Done begin upload segment. Dec 30 11:05:31.321897207 (595h36m31.192980242s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.322782493 (595h36m31.193865538s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323184989 (595h36m31.194268024s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323494158 (595h36m31.194577193s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323955717 (595h36m31.195038792s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324400043 (595h36m31.195483088s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324830674 (595h36m31.195913719s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325147958 (595h36m31.196230993s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325505809 (595h36m31.196588844s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.326834970 (595h36m31.197918005s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.328334906 (595h36m31.199417941s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.329770349 (595h36m31.200853394s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.331418517 (595h36m31.202501562s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.332734062 (595h36m31.203817097s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.333628375 (595h36m31.20471142s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.334856594 (595h36m31.205939639s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:33.040067914 (595h36m32.911150959s) [upload=522119]: Next returned nil segment Dec 30 11:05:33.040080628 (595h36m32.911163673s) [upload=522119]: Waiting for error group managing segments... Dec 30 11:05:33.318845022 (595h36m33.189928057s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.324469852 (595h36m33.195552887s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.334622755 (595h36m33.20570579s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.335109842 (595h36m33.206192887s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337738137 (595h36m33.208821182s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337942486 (595h36m33.209025531s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.343300238 (595h36m33.214383283s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.408823275 (595h36m33.27990632s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.600573588 (595h36m33.471656633s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.639243532 (595h36m33.510326567s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.646759033 (595h36m33.517842078s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.678953487 (595h36m33.550036532s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684164679 (595h36m33.555247714s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684188575 (595h36m33.55527161s) [seg_pos={0 0}, upload=522119]: Segment reached optimal threshold of 13 pieces. Dec 30 11:05:33.684483066 (595h36m33.555566101s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684491702 (595h36m33.555574737s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684811281 (595h36m33.555894316s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684819656 (595h36m33.555902701s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684956717 (595h36m33.556039762s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684963410 (595h36m33.556046445s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684981445 (595h36m33.55606448s) [seg_pos={0 0}, upload=522119]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:05:33.685058802 (595h36m33.556141847s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:05:33.685064963 (595h36m33.556147998s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:05:33.979160510 (595h36m33.850243545s) [upload=522119]: Done uploading segments... Dec 30 11:05:33.979184876 (595h36m33.850267921s) [upload=522119]: Upload finished. err: <nil> Duration: 6.016630554s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.........X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.X] duration: 5.00043379s trace: https://jaeger.dev.storj.io/trace/1e8ce1f3be3ecd1 FAIL (2025-12-30 11:06:47 UTC) orbiter warning: context deadline exceeded Duration: 5.000446053s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..........X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X.] duration: 3.624575904s trace: https://jaeger.dev.storj.io/trace/47b5affd61422eaa FAIL (2025-12-30 11:06:14 UTC) orbiter warning: context deadline exceeded Duration: 3.624713336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....XX...X.X..XX.XX........X.......X..XX..X.X.X..........X.] duration: 5.776979366s trace: https://jaeger.dev.storj.io/trace/1efd62034a49d3b FAIL (2025-12-30 11:06:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ) context deadline exceeded ======LOGS====== Dec 30 11:05:31.076907427 (595h36m30.947990462s) [upload=522119]: Starting upload Dec 30 11:05:31.076967993 (595h36m30.948051038s) [upload=522119]: Done starting upload Dec 30 11:05:31.077053936 (595h36m30.948136971s) [upload=522119]: Uploading segments... Dec 30 11:05:31.077197178 (595h36m30.948280213s) [upload=522119]: Got next segment. Inline: false Dec 30 11:05:31.077233808 (595h36m30.948316843s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:05:31.077238317 (595h36m30.948321352s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:05:31.321780635 (595h36m31.19286368s) [seg_pos={0 0}, upload=522119]: Begin upload segment... Dec 30 11:05:31.321866699 (595h36m31.192949744s) [seg_pos={0 0}, upload=522119]: Done begin upload segment. Dec 30 11:05:31.321897207 (595h36m31.192980242s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.322782493 (595h36m31.193865538s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323184989 (595h36m31.194268024s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323494158 (595h36m31.194577193s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323955717 (595h36m31.195038792s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324400043 (595h36m31.195483088s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324830674 (595h36m31.195913719s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325147958 (595h36m31.196230993s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325505809 (595h36m31.196588844s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.326834970 (595h36m31.197918005s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.328334906 (595h36m31.199417941s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.329770349 (595h36m31.200853394s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.331418517 (595h36m31.202501562s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.332734062 (595h36m31.203817097s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.333628375 (595h36m31.20471142s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.334856594 (595h36m31.205939639s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:33.040067914 (595h36m32.911150959s) [upload=522119]: Next returned nil segment Dec 30 11:05:33.040080628 (595h36m32.911163673s) [upload=522119]: Waiting for error group managing segments... Dec 30 11:05:33.318845022 (595h36m33.189928057s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.324469852 (595h36m33.195552887s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.334622755 (595h36m33.20570579s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.335109842 (595h36m33.206192887s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337738137 (595h36m33.208821182s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337942486 (595h36m33.209025531s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.343300238 (595h36m33.214383283s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.408823275 (595h36m33.27990632s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.600573588 (595h36m33.471656633s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.639243532 (595h36m33.510326567s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.646759033 (595h36m33.517842078s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.678953487 (595h36m33.550036532s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684164679 (595h36m33.555247714s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684188575 (595h36m33.55527161s) [seg_pos={0 0}, upload=522119]: Segment reached optimal threshold of 13 pieces. Dec 30 11:05:33.684483066 (595h36m33.555566101s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684491702 (595h36m33.555574737s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684811281 (595h36m33.555894316s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684819656 (595h36m33.555902701s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684956717 (595h36m33.556039762s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684963410 (595h36m33.556046445s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684981445 (595h36m33.55606448s) [seg_pos={0 0}, upload=522119]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:05:33.685058802 (595h36m33.556141847s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:05:33.685064963 (595h36m33.556147998s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:05:33.979160510 (595h36m33.850243545s) [upload=522119]: Done uploading segments... Dec 30 11:05:33.979184876 (595h36m33.850267921s) [upload=522119]: Upload finished. err: <nil> Duration: 5.777088864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......XX...X.X..XX.XX........X.......X..XX..X.X.X..........X] duration: 30.004417121s trace: https://jaeger.dev.storj.io/trace/1efd62034a49d3b FAIL (2025-12-30 11:06:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ) context deadline exceeded ======LOGS====== Dec 30 11:05:31.076907427 (595h36m30.947990462s) [upload=522119]: Starting upload Dec 30 11:05:31.076967993 (595h36m30.948051038s) [upload=522119]: Done starting upload Dec 30 11:05:31.077053936 (595h36m30.948136971s) [upload=522119]: Uploading segments... Dec 30 11:05:31.077197178 (595h36m30.948280213s) [upload=522119]: Got next segment. Inline: false Dec 30 11:05:31.077233808 (595h36m30.948316843s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 11:05:31.077238317 (595h36m30.948321352s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 11:05:31.321780635 (595h36m31.19286368s) [seg_pos={0 0}, upload=522119]: Begin upload segment... Dec 30 11:05:31.321866699 (595h36m31.192949744s) [seg_pos={0 0}, upload=522119]: Done begin upload segment. Dec 30 11:05:31.321897207 (595h36m31.192980242s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.322782493 (595h36m31.193865538s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323184989 (595h36m31.194268024s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323494158 (595h36m31.194577193s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.323955717 (595h36m31.195038792s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324400043 (595h36m31.195483088s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.324830674 (595h36m31.195913719s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325147958 (595h36m31.196230993s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.325505809 (595h36m31.196588844s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.326834970 (595h36m31.197918005s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.328334906 (595h36m31.199417941s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.329770349 (595h36m31.200853394s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.331418517 (595h36m31.202501562s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.332734062 (595h36m31.203817097s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.333628375 (595h36m31.20471142s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:31.334856594 (595h36m31.205939639s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Uploading piece... Dec 30 11:05:33.040067914 (595h36m32.911150959s) [upload=522119]: Next returned nil segment Dec 30 11:05:33.040080628 (595h36m32.911163673s) [upload=522119]: Waiting for error group managing segments... Dec 30 11:05:33.318845022 (595h36m33.189928057s) [noise=true , address=145.79.227.9:4367 , piece_id=LZ4TPC3KNIW2GIRJM52PI66ECBFZHLMYBNX7T2OFX24QGIFI7A6A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.324469852 (595h36m33.195552887s) [noise=true , address=145.79.227.37:4567 , piece_id=YR6AYVOSYGKTRVLLGT5W4JBERIQTX3VYGBC27ZG3CAVB6EYI7ITQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.334622755 (595h36m33.20570579s) [noise=true , address=145.79.227.19:4077 , piece_id=5YIMWG5U7U3YA6WAJBTY5JO4TSFQDU2GBKLBBZLHOOKVFQZUUY2A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.335109842 (595h36m33.206192887s) [noise=true , address=145.79.227.16:4909 , piece_id=FDQQSTIGXK7427LMKVSDPI6RIRSJBGY3CQJXV4A66RGJW2PLAAMA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337738137 (595h36m33.208821182s) [noise=true , address=145.79.227.15:4876 , piece_id=T3RJZRXGQQXCDZLL42NKXZXLQ4KWKI2ESTDEDNZ6MUHY7COJR4NQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.337942486 (595h36m33.209025531s) [noise=true , address=145.79.227.11:4730 , piece_id=R3CUCOAF6Z5X2LKYXKV7ZOL73LWKWCXYGESTITE3UBJTXGIGJIXQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.343300238 (595h36m33.214383283s) [noise=true , address=145.79.227.23:4547 , piece_id=RGHYW75X5MNTA33RZS46OBRLIQSVEYS63US6PADZT3GQN3LHBPRQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.408823275 (595h36m33.27990632s) [noise=true , address=145.79.227.17:4972 , piece_id=G4MBK423NIZBCGL72Q7ST5T3KO6HA76UBUGFXHOZZ3BB3VN3KQWA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.600573588 (595h36m33.471656633s) [noise=true , address=77.90.31.13:4627 , piece_id=DNK5ZNEFMJFZZ4GZUYLF7CG4SNFOZWEK7JMHNAG2VUMV3TZX7NGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.639243532 (595h36m33.510326567s) [noise=true , address=45.91.116.4:4937 , piece_id=52BKXWCZS5NPUCSG7JJZKX5XY6MILG234WEGQCFQ74JVBP4COV7Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.646759033 (595h36m33.517842078s) [noise=true , address=45.91.116.14:4192 , piece_id=P76CZCPT36FP4HGL7TFEOR7X3E6YH6KKYJH5BKO3FHADGNRLFT5Q, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.678953487 (595h36m33.550036532s) [noise=true , address=45.91.116.8:4045 , piece_id=ISGQ7A2VBYALVMIUCEJXIBVMPVMPEW4CCSFXMBEZDMHRJAKABTJQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684164679 (595h36m33.555247714s) [noise=true , address=45.91.116.11:4119 , piece_id=M6YB3NVKD27T6747WDEG524TILZBIDYSASP2R4ZB5IQMO737GXHQ, seg_pos={0 0}, upload=522119]: Done uploading piece. err: <nil> Dec 30 11:05:33.684188575 (595h36m33.55527161s) [seg_pos={0 0}, upload=522119]: Segment reached optimal threshold of 13 pieces. Dec 30 11:05:33.684483066 (595h36m33.555566101s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684491702 (595h36m33.555574737s) [noise=true , address=45.91.116.12:4226 , piece_id=DEEUDTDECK5JADHZKL4YX2G3P6RPON5ZDIGVMMRBJCJTZU2FAB7A, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684811281 (595h36m33.555894316s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684819656 (595h36m33.555902701s) [noise=true , address=45.91.116.10:4084 , piece_id=X73XVSU2LBBYLHNDJCJGV5SYZJLQC42LPCTRYFHGPYVTUANIIVGA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684956717 (595h36m33.556039762s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 30 11:05:33.684963410 (595h36m33.556046445s) [noise=true , address=45.91.116.9:4264 , piece_id=6QNIZ7WWPTW2NDEBJIVCKQREVBIHP2QOCBR4BLEEKIPPLL3DNRJA, seg_pos={0 0}, upload=522119]: Long tail cancellation due to optimal threshold Dec 30 11:05:33.684981445 (595h36m33.55606448s) [seg_pos={0 0}, upload=522119]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 11:05:33.685058802 (595h36m33.556141847s) [upload=522119]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 11:05:33.685064963 (595h36m33.556147998s) [upload=522119]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 11:05:33.979160510 (595h36m33.850243545s) [upload=522119]: Done uploading segments... Dec 30 11:05:33.979184876 (595h36m33.850267921s) [upload=522119]: Upload finished. err: <nil> Duration: 30.004441369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.........X....................X.................X.......] duration: 34.022354032s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:05:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 34.022367579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [...........X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.X] duration: 5.001430419s trace: https://jaeger.dev.storj.io/trace/47b5affd61422eaa FAIL (2025-12-30 11:05:30 UTC) orbiter warning: context deadline exceeded Duration: 5.0014519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.........X....................X.................X......] duration: 45.044859088s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:05:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 45.044886579s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [............X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX.] duration: 3.693341323s trace: https://jaeger.dev.storj.io/trace/515de62355c64bab FAIL (2025-12-30 11:04:49 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.693421207s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......XX...X.X..XX.XX........X.......X..XX..X.X.X..........] duration: 5.826838559s trace: https://jaeger.dev.storj.io/trace/4462c9db966cc2a FAIL (2025-12-30 11:04:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA) context deadline exceeded ======LOGS====== 7 (595h26m47.753396192s) [upload=521980]: Done starting upload Dec 30 10:55:47.882409701 (595h26m47.753492746s) [upload=521980]: Uploading segments... Dec 30 10:55:47.882622086 (595h26m47.753705141s) [upload=521980]: Got next segment. Inline: false Dec 30 10:55:47.882677732 (595h26m47.753760777s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:55:47.882684334 (595h26m47.753767379s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:55:48.132811478 (595h26m48.003894513s) [seg_pos={0 0}, upload=521980]: Begin upload segment... Dec 30 10:55:48.132886200 (595h26m48.003969245s) [seg_pos={0 0}, upload=521980]: Done begin upload segment. Dec 30 10:55:48.132935223 (595h26m48.004018258s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133006459 (595h26m48.004089494s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133505349 (595h26m48.004588394s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133867649 (595h26m48.004950694s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134246290 (595h26m48.005329335s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134718358 (595h26m48.005801404s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134724330 (595h26m48.005807365s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135049399 (595h26m48.006132434s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135164258 (595h26m48.006247293s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135396590 (595h26m48.006479626s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135505588 (595h26m48.006588633s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135738722 (595h26m48.006821767s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135845255 (595h26m48.0069283s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136209859 (595h26m48.007292904s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136633235 (595h26m48.00771627s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.137244088 (595h26m48.008327123s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:49.839444246 (595h26m49.710527281s) [upload=521980]: Next returned nil segment Dec 30 10:55:49.839461207 (595h26m49.710544273s) [upload=521980]: Waiting for error group managing segments... Dec 30 10:55:50.117467287 (595h26m49.988550332s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.117565533 (595h26m49.988648578s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.122601262 (595h26m49.993684307s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.124079047 (595h26m49.995162082s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.129932483 (595h26m50.001015528s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.133574788 (595h26m50.004657823s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.138294364 (595h26m50.009377399s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.145152304 (595h26m50.016235339s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.148344040 (595h26m50.019427086s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.209083830 (595h26m50.080166865s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.507112905 (595h26m50.37819595s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.692343985 (595h26m50.56342703s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290837664 (595h26m51.16192072s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290862201 (595h26m51.161945257s) [seg_pos={0 0}, upload=521980]: Segment reached optimal threshold of 13 pieces. Dec 30 10:55:51.291085287 (595h26m51.162168352s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 10:55:51.291094224 (595h26m51.162177269s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291211416 (595h26m51.162294471s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:36672->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 10:55:51.291222287 (595h26m51.162305343s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291239801 (595h26m51.162322866s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection Dec 30 10:55:51.291248657 (595h26m51.162331702s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291276230 (595h26m51.162359275s) [seg_pos={0 0}, upload=521980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:55:51.291468056 (595h26m51.162551091s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:55:51.291477233 (595h26m51.162560278s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:55:51.589238178 (595h26m51.460321224s) [upload=521980]: Done uploading segments... Dec 30 10:55:51.589255381 (595h26m51.460338416s) [upload=521980]: Upload finished. err: <nil> Duration: 5.826865621s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [X....X.........X....................X.................X.....] duration: 21.086496943s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:04:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 21.086527282s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X............X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.XX] duration: 5.001084377s trace: https://jaeger.dev.storj.io/trace/515de62355c64bab FAIL (2025-12-30 11:03:53 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001099715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X....X.........X....................X.................X....] duration: 36.302864128s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:03:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 36.302881922s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........XX...X.X..XX.XX........X.......X..XX..X.X.X.........] duration: 21.662939045s trace: https://jaeger.dev.storj.io/trace/4462c9db966cc2a FAIL (2025-12-30 11:03:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA) context deadline exceeded ======LOGS====== 7 (595h26m47.753396192s) [upload=521980]: Done starting upload Dec 30 10:55:47.882409701 (595h26m47.753492746s) [upload=521980]: Uploading segments... Dec 30 10:55:47.882622086 (595h26m47.753705141s) [upload=521980]: Got next segment. Inline: false Dec 30 10:55:47.882677732 (595h26m47.753760777s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:55:47.882684334 (595h26m47.753767379s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:55:48.132811478 (595h26m48.003894513s) [seg_pos={0 0}, upload=521980]: Begin upload segment... Dec 30 10:55:48.132886200 (595h26m48.003969245s) [seg_pos={0 0}, upload=521980]: Done begin upload segment. Dec 30 10:55:48.132935223 (595h26m48.004018258s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133006459 (595h26m48.004089494s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133505349 (595h26m48.004588394s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133867649 (595h26m48.004950694s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134246290 (595h26m48.005329335s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134718358 (595h26m48.005801404s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134724330 (595h26m48.005807365s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135049399 (595h26m48.006132434s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135164258 (595h26m48.006247293s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135396590 (595h26m48.006479626s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135505588 (595h26m48.006588633s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135738722 (595h26m48.006821767s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135845255 (595h26m48.0069283s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136209859 (595h26m48.007292904s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136633235 (595h26m48.00771627s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.137244088 (595h26m48.008327123s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:49.839444246 (595h26m49.710527281s) [upload=521980]: Next returned nil segment Dec 30 10:55:49.839461207 (595h26m49.710544273s) [upload=521980]: Waiting for error group managing segments... Dec 30 10:55:50.117467287 (595h26m49.988550332s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.117565533 (595h26m49.988648578s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.122601262 (595h26m49.993684307s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.124079047 (595h26m49.995162082s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.129932483 (595h26m50.001015528s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.133574788 (595h26m50.004657823s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.138294364 (595h26m50.009377399s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.145152304 (595h26m50.016235339s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.148344040 (595h26m50.019427086s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.209083830 (595h26m50.080166865s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.507112905 (595h26m50.37819595s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.692343985 (595h26m50.56342703s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290837664 (595h26m51.16192072s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290862201 (595h26m51.161945257s) [seg_pos={0 0}, upload=521980]: Segment reached optimal threshold of 13 pieces. Dec 30 10:55:51.291085287 (595h26m51.162168352s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 10:55:51.291094224 (595h26m51.162177269s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291211416 (595h26m51.162294471s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:36672->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 10:55:51.291222287 (595h26m51.162305343s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291239801 (595h26m51.162322866s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection Dec 30 10:55:51.291248657 (595h26m51.162331702s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291276230 (595h26m51.162359275s) [seg_pos={0 0}, upload=521980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:55:51.291468056 (595h26m51.162551091s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:55:51.291477233 (595h26m51.162560278s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:55:51.589238178 (595h26m51.460321224s) [upload=521980]: Done uploading segments... Dec 30 10:55:51.589255381 (595h26m51.460338416s) [upload=521980]: Upload finished. err: <nil> Duration: 21.662972648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X....X.........X....................X.................X...] duration: 35.107747163s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:03:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 35.107779794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.X............X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.X] duration: 5.001202854s trace: https://jaeger.dev.storj.io/trace/63ffce7d0eaf3983 FAIL (2025-12-30 11:03:02 UTC) orbiter warning: context deadline exceeded Duration: 5.001232341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........XX...X.X..XX.XX........X.......X..XX..X.X.X........] duration: 5.985246184s trace: https://jaeger.dev.storj.io/trace/4462c9db966cc2a FAIL (2025-12-30 11:02:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA) context deadline exceeded ======LOGS====== 7 (595h26m47.753396192s) [upload=521980]: Done starting upload Dec 30 10:55:47.882409701 (595h26m47.753492746s) [upload=521980]: Uploading segments... Dec 30 10:55:47.882622086 (595h26m47.753705141s) [upload=521980]: Got next segment. Inline: false Dec 30 10:55:47.882677732 (595h26m47.753760777s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:55:47.882684334 (595h26m47.753767379s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:55:48.132811478 (595h26m48.003894513s) [seg_pos={0 0}, upload=521980]: Begin upload segment... Dec 30 10:55:48.132886200 (595h26m48.003969245s) [seg_pos={0 0}, upload=521980]: Done begin upload segment. Dec 30 10:55:48.132935223 (595h26m48.004018258s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133006459 (595h26m48.004089494s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133505349 (595h26m48.004588394s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133867649 (595h26m48.004950694s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134246290 (595h26m48.005329335s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134718358 (595h26m48.005801404s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134724330 (595h26m48.005807365s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135049399 (595h26m48.006132434s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135164258 (595h26m48.006247293s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135396590 (595h26m48.006479626s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135505588 (595h26m48.006588633s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135738722 (595h26m48.006821767s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135845255 (595h26m48.0069283s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136209859 (595h26m48.007292904s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136633235 (595h26m48.00771627s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.137244088 (595h26m48.008327123s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:49.839444246 (595h26m49.710527281s) [upload=521980]: Next returned nil segment Dec 30 10:55:49.839461207 (595h26m49.710544273s) [upload=521980]: Waiting for error group managing segments... Dec 30 10:55:50.117467287 (595h26m49.988550332s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.117565533 (595h26m49.988648578s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.122601262 (595h26m49.993684307s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.124079047 (595h26m49.995162082s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.129932483 (595h26m50.001015528s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.133574788 (595h26m50.004657823s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.138294364 (595h26m50.009377399s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.145152304 (595h26m50.016235339s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.148344040 (595h26m50.019427086s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.209083830 (595h26m50.080166865s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.507112905 (595h26m50.37819595s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.692343985 (595h26m50.56342703s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290837664 (595h26m51.16192072s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290862201 (595h26m51.161945257s) [seg_pos={0 0}, upload=521980]: Segment reached optimal threshold of 13 pieces. Dec 30 10:55:51.291085287 (595h26m51.162168352s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 10:55:51.291094224 (595h26m51.162177269s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291211416 (595h26m51.162294471s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:36672->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 10:55:51.291222287 (595h26m51.162305343s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291239801 (595h26m51.162322866s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection Dec 30 10:55:51.291248657 (595h26m51.162331702s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291276230 (595h26m51.162359275s) [seg_pos={0 0}, upload=521980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:55:51.291468056 (595h26m51.162551091s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:55:51.291477233 (595h26m51.162560278s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:55:51.589238178 (595h26m51.460321224s) [upload=521980]: Done uploading segments... Dec 30 10:55:51.589255381 (595h26m51.460338416s) [upload=521980]: Upload finished. err: <nil> Duration: 5.985273245s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X....X.........X....................X.................X..] duration: 44.20700385s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:02:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 44.207028999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [X.X............X......X.X.XXXX.X.XX.X.X.....XX.X........X.X.] duration: 3.471651009s trace: https://jaeger.dev.storj.io/trace/60fd4a5b2a8e2a00 FAIL (2025-12-30 11:02:15 UTC) orbiter warning: context deadline exceeded Duration: 3.47168386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..........XX...X.X..XX.XX........X.......X..XX..X.X.X.......] duration: 5.876809871s trace: https://jaeger.dev.storj.io/trace/4462c9db966cc2a FAIL (2025-12-30 11:01:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA) context deadline exceeded ======LOGS====== 7 (595h26m47.753396192s) [upload=521980]: Done starting upload Dec 30 10:55:47.882409701 (595h26m47.753492746s) [upload=521980]: Uploading segments... Dec 30 10:55:47.882622086 (595h26m47.753705141s) [upload=521980]: Got next segment. Inline: false Dec 30 10:55:47.882677732 (595h26m47.753760777s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:55:47.882684334 (595h26m47.753767379s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:55:48.132811478 (595h26m48.003894513s) [seg_pos={0 0}, upload=521980]: Begin upload segment... Dec 30 10:55:48.132886200 (595h26m48.003969245s) [seg_pos={0 0}, upload=521980]: Done begin upload segment. Dec 30 10:55:48.132935223 (595h26m48.004018258s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133006459 (595h26m48.004089494s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133505349 (595h26m48.004588394s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133867649 (595h26m48.004950694s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134246290 (595h26m48.005329335s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134718358 (595h26m48.005801404s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134724330 (595h26m48.005807365s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135049399 (595h26m48.006132434s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135164258 (595h26m48.006247293s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135396590 (595h26m48.006479626s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135505588 (595h26m48.006588633s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135738722 (595h26m48.006821767s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135845255 (595h26m48.0069283s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136209859 (595h26m48.007292904s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136633235 (595h26m48.00771627s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.137244088 (595h26m48.008327123s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:49.839444246 (595h26m49.710527281s) [upload=521980]: Next returned nil segment Dec 30 10:55:49.839461207 (595h26m49.710544273s) [upload=521980]: Waiting for error group managing segments... Dec 30 10:55:50.117467287 (595h26m49.988550332s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.117565533 (595h26m49.988648578s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.122601262 (595h26m49.993684307s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.124079047 (595h26m49.995162082s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.129932483 (595h26m50.001015528s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.133574788 (595h26m50.004657823s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.138294364 (595h26m50.009377399s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.145152304 (595h26m50.016235339s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.148344040 (595h26m50.019427086s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.209083830 (595h26m50.080166865s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.507112905 (595h26m50.37819595s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.692343985 (595h26m50.56342703s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290837664 (595h26m51.16192072s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290862201 (595h26m51.161945257s) [seg_pos={0 0}, upload=521980]: Segment reached optimal threshold of 13 pieces. Dec 30 10:55:51.291085287 (595h26m51.162168352s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 10:55:51.291094224 (595h26m51.162177269s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291211416 (595h26m51.162294471s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:36672->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 10:55:51.291222287 (595h26m51.162305343s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291239801 (595h26m51.162322866s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection Dec 30 10:55:51.291248657 (595h26m51.162331702s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291276230 (595h26m51.162359275s) [seg_pos={0 0}, upload=521980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:55:51.291468056 (595h26m51.162551091s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:55:51.291477233 (595h26m51.162560278s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:55:51.589238178 (595h26m51.460321224s) [upload=521980]: Done uploading segments... Dec 30 10:55:51.589255381 (595h26m51.460338416s) [upload=521980]: Upload finished. err: <nil> Duration: 5.876862851s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.X.X............X......X.X.XXXX.X.XX.X.X.....XX.X........X.X] duration: 5.000854181s trace: https://jaeger.dev.storj.io/trace/60fd4a5b2a8e2a00 FAIL (2025-12-30 11:01:22 UTC) orbiter warning: context deadline exceeded Duration: 5.000871033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X....X.........X....................X.................X.] duration: 36.560528276s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:00:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 36.560560537s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..X.X............X......X.X.XXXX.X.XX.X.X.....XX.X........X.] duration: 3.417482093s trace: https://jaeger.dev.storj.io/trace/1582f43044e1cf55 FAIL (2025-12-30 11:00:51 UTC) orbiter warning: context deadline exceeded Duration: 3.41750709s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..........XX...X.X..XX.XX........X.......X..XX..X.X.X......] duration: 8.454099243s trace: https://jaeger.dev.storj.io/trace/4462c9db966cc2a FAIL (2025-12-30 11:00:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA) context deadline exceeded ======LOGS====== 7 (595h26m47.753396192s) [upload=521980]: Done starting upload Dec 30 10:55:47.882409701 (595h26m47.753492746s) [upload=521980]: Uploading segments... Dec 30 10:55:47.882622086 (595h26m47.753705141s) [upload=521980]: Got next segment. Inline: false Dec 30 10:55:47.882677732 (595h26m47.753760777s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:55:47.882684334 (595h26m47.753767379s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:55:48.132811478 (595h26m48.003894513s) [seg_pos={0 0}, upload=521980]: Begin upload segment... Dec 30 10:55:48.132886200 (595h26m48.003969245s) [seg_pos={0 0}, upload=521980]: Done begin upload segment. Dec 30 10:55:48.132935223 (595h26m48.004018258s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133006459 (595h26m48.004089494s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133505349 (595h26m48.004588394s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133867649 (595h26m48.004950694s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134246290 (595h26m48.005329335s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134718358 (595h26m48.005801404s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134724330 (595h26m48.005807365s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135049399 (595h26m48.006132434s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135164258 (595h26m48.006247293s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135396590 (595h26m48.006479626s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135505588 (595h26m48.006588633s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135738722 (595h26m48.006821767s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135845255 (595h26m48.0069283s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136209859 (595h26m48.007292904s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136633235 (595h26m48.00771627s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.137244088 (595h26m48.008327123s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:49.839444246 (595h26m49.710527281s) [upload=521980]: Next returned nil segment Dec 30 10:55:49.839461207 (595h26m49.710544273s) [upload=521980]: Waiting for error group managing segments... Dec 30 10:55:50.117467287 (595h26m49.988550332s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.117565533 (595h26m49.988648578s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.122601262 (595h26m49.993684307s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.124079047 (595h26m49.995162082s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.129932483 (595h26m50.001015528s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.133574788 (595h26m50.004657823s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.138294364 (595h26m50.009377399s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.145152304 (595h26m50.016235339s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.148344040 (595h26m50.019427086s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.209083830 (595h26m50.080166865s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.507112905 (595h26m50.37819595s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.692343985 (595h26m50.56342703s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290837664 (595h26m51.16192072s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290862201 (595h26m51.161945257s) [seg_pos={0 0}, upload=521980]: Segment reached optimal threshold of 13 pieces. Dec 30 10:55:51.291085287 (595h26m51.162168352s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 10:55:51.291094224 (595h26m51.162177269s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291211416 (595h26m51.162294471s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:36672->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 10:55:51.291222287 (595h26m51.162305343s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291239801 (595h26m51.162322866s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection Dec 30 10:55:51.291248657 (595h26m51.162331702s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291276230 (595h26m51.162359275s) [seg_pos={0 0}, upload=521980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:55:51.291468056 (595h26m51.162551091s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:55:51.291477233 (595h26m51.162560278s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:55:51.589238178 (595h26m51.460321224s) [upload=521980]: Done uploading segments... Dec 30 10:55:51.589255381 (595h26m51.460338416s) [upload=521980]: Upload finished. err: <nil> Duration: 8.45412366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X....X.........X....................X.................X] duration: 1m0.005313785s trace: https://jaeger.dev.storj.io/trace/64cf22e06831b0c1 FAIL (2025-12-30 11:00:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 0m8.982649071s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.113681765 (595h30m8.98476481s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.115636917 (595h30m8.986719962s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.116718737 (595h30m8.987801782s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.117436585 (595h30m8.98851963s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.118590522 (595h30m8.989673567s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.119451251 (595h30m8.990534296s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.120295318 (595h30m8.991378363s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.121311162 (595h30m8.992394207s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.123321070 (595h30m8.994404115s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.125543381 (595h30m8.996626427s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.129711698 (595h30m9.000794743s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.135978491 (595h30m9.007061526s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.147053721 (595h30m9.018136766s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.150314600 (595h30m9.021397645s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:09.154924809 (595h30m9.026007854s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Uploading piece... Dec 30 10:59:10.552042258 (595h30m10.423125293s) [upload=522027]: Next returned nil segment Dec 30 10:59:10.552054110 (595h30m10.423137145s) [upload=522027]: Waiting for error group managing segments... Dec 30 10:59:11.004070252 (595h30m10.875153307s) [noise=true , address=145.79.227.6:4273 , piece_id=EFSU5TKE3XWGIVPIVE5FH2CY3L7ZJW4KNWBM7QEIDOBQJJYCEHZA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.006048319 (595h30m10.877131354s) [noise=true , address=145.79.227.16:4917 , piece_id=AUSCRBBGBE2RUAEXYJVVPQQMOW2J2IPGBWFK6GL5GQMLE4LWUQVA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.025885832 (595h30m10.896968877s) [noise=true , address=145.79.227.14:4856 , piece_id=QUDTKLMF2ZVPVAKU5OIROP7WI73DLKWVCIUC4ZTI7D6NXI2FTALQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.030546126 (595h30m10.901629161s) [noise=true , address=145.79.227.4:4211 , piece_id=XIMHSAGR5W5FPFOTY3TMCHKRWW22V5UODXWYQ56JOQDOAZV6POEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.031271938 (595h30m10.902354973s) [noise=true , address=145.79.227.19:4090 , piece_id=DXUFOJI2LEPP53KUDO2IYLRNMRR3R6ZBH2O6IL4RBQNTIKMGTEAQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.268614337 (595h30m11.139697382s) [noise=true , address=145.79.227.35:4508 , piece_id=5WGTIDSOJE6FEBF45QLQ36CP5EAQCG35VYWTYM2VYK6AJR2SALPQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.304422241 (595h30m11.175505286s) [noise=true , address=77.90.31.18:4817 , piece_id=QJC27EVDL6N3OXBKZMBNHOBTX7ZR3VOHDMXCZ5KT74MSXG5EYV2Q, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.895686774 (595h30m11.76676983s) [noise=true , address=77.90.31.21:4897 , piece_id=4UOWECWZROEJ3DKULGTBN6ZNAGTTJHYPMU7YFJ6I44BU66UANOFA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:11.967521580 (595h30m11.838604615s) [noise=true , address=145.79.227.30:4770 , piece_id=OAE43JVU26ILIF7DL3PFGIKGCVVXMRJB3J4BY3M46YZKPW5DWFBA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.005421898 (595h30m11.876504963s) [noise=true , address=145.79.227.44:4200 , piece_id=MXG3RZPJZLLBXXGMTXOYAX6SIDKQVN24SVPFRL4NTK43CWDTPHCQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.020264291 (595h30m11.891347326s) [noise=true , address=77.90.31.16:4740 , piece_id=FXBHUH3KFBA5U2O5HZX7CFPUKYXVRYIMLQROTKRNAE4X5O54AZ7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.054845209 (595h30m11.925928244s) [noise=true , address=193.149.164.143:4699 , piece_id=ZOTTV46CECHNMIPNSEI2UXTBORWFN22YA3D3R6CXAZLHBF2VHT6A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.082294416 (595h30m11.953377461s) [noise=true , address=145.79.227.49:4370 , piece_id=SCBOG6IBTQA7QNL72LZ6Q7DKLTCKR7XXXS3FVJC2OTC7AML7AP7A, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153131852 (595h30m12.024214897s) [noise=true , address=45.91.116.7:4041 , piece_id=6WN4PTRGIW2OVJMW3OO5D33T4LPBYRFCJWZJELLOLIDZKXBA5QJQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: <nil> Dec 30 10:59:12.153161859 (595h30m12.024244894s) [seg_pos={0 0}, upload=522027]: Segment reached optimal threshold of 14 pieces. Dec 30 10:59:12.153538827 (595h30m12.024621862s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FkuTqtaF3DmzHFvcCuJRvaubQV6HbaV1CnFtoY95QHH6NcME): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153550519 (595h30m12.024633584s) [noise=true , address=145.79.227.33:4888 , piece_id=BT6WGUZWQRJXAXHYAQ46B2TB6YK37YXQF6ZBIDYCCUSVZQE3MMQA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153555559 (595h30m12.024638605s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RYKzn5zVSmxVbWwnBdX8oqtG1WE2MhPeEiWLf5dZeFAut6s2Y): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153572191 (595h30m12.024655236s) [noise=true , address=145.79.227.40:4079 , piece_id=ARIO5ODSQUW4XYKN6VQN6F2IXK3FMZBG3AV2DVXKENIPBE435UEQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153752013 (595h30m12.024835058s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1okyMVGQotKJf1ysn8KeBxsmQJoR1amH1kLtYpgYGnADRcK5C8): protocol: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36910->64.49.13.26:4952: use of closed network connection Dec 30 10:59:12.153758184 (595h30m12.02484123s) [noise=true , address=64.49.13.26:4952 , piece_id=HDKLN5OPNPM3VCZ5B5G2MNWYGSPLOJEYJ5QIK7EJTHCB3JGRP7DA, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153776680 (595h30m12.024859745s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 30 10:59:12.153787350 (595h30m12.024870395s) [noise=true , address=145.79.227.34:4913 , piece_id=YDHK4XVB6S5RJG74YQEIVGPFTMKCA2L27L2XFB6N6YMPJJW3QIDQ, seg_pos={0 0}, upload=522027]: Long tail cancellation due to optimal threshold Dec 30 10:59:12.153885758 (595h30m12.024968803s) [seg_pos={0 0}, upload=522027]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:59:12.153983694 (595h30m12.025066729s) [upload=522027]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:59:12.153992220 (595h30m12.025075275s) [upload=522027]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:59:12.448755408 (595h30m12.319838453s) [upload=522027]: Done uploading segments... Dec 30 10:59:12.448781498 (595h30m12.319864543s) [upload=522027]: Upload finished. err: <nil> Duration: 1m0.005324896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...X.X............X......X.X.XXXX.X.XX.X.X.....XX.X........X] duration: 5.000767187s trace: https://jaeger.dev.storj.io/trace/1582f43044e1cf55 FAIL (2025-12-30 10:59:49 UTC) orbiter warning: context deadline exceeded Duration: 5.000783407s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX..........XX...X.X..XX.XX........X.......X..XX..X.X.X.....] duration: 5.979011162s trace: https://jaeger.dev.storj.io/trace/4462c9db966cc2a FAIL (2025-12-30 10:59:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA) context deadline exceeded ======LOGS====== 7 (595h26m47.753396192s) [upload=521980]: Done starting upload Dec 30 10:55:47.882409701 (595h26m47.753492746s) [upload=521980]: Uploading segments... Dec 30 10:55:47.882622086 (595h26m47.753705141s) [upload=521980]: Got next segment. Inline: false Dec 30 10:55:47.882677732 (595h26m47.753760777s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:55:47.882684334 (595h26m47.753767379s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:55:48.132811478 (595h26m48.003894513s) [seg_pos={0 0}, upload=521980]: Begin upload segment... Dec 30 10:55:48.132886200 (595h26m48.003969245s) [seg_pos={0 0}, upload=521980]: Done begin upload segment. Dec 30 10:55:48.132935223 (595h26m48.004018258s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133006459 (595h26m48.004089494s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133505349 (595h26m48.004588394s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133867649 (595h26m48.004950694s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134246290 (595h26m48.005329335s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134718358 (595h26m48.005801404s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134724330 (595h26m48.005807365s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135049399 (595h26m48.006132434s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135164258 (595h26m48.006247293s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135396590 (595h26m48.006479626s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135505588 (595h26m48.006588633s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135738722 (595h26m48.006821767s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135845255 (595h26m48.0069283s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136209859 (595h26m48.007292904s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136633235 (595h26m48.00771627s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.137244088 (595h26m48.008327123s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:49.839444246 (595h26m49.710527281s) [upload=521980]: Next returned nil segment Dec 30 10:55:49.839461207 (595h26m49.710544273s) [upload=521980]: Waiting for error group managing segments... Dec 30 10:55:50.117467287 (595h26m49.988550332s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.117565533 (595h26m49.988648578s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.122601262 (595h26m49.993684307s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.124079047 (595h26m49.995162082s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.129932483 (595h26m50.001015528s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.133574788 (595h26m50.004657823s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.138294364 (595h26m50.009377399s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.145152304 (595h26m50.016235339s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.148344040 (595h26m50.019427086s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.209083830 (595h26m50.080166865s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.507112905 (595h26m50.37819595s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.692343985 (595h26m50.56342703s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290837664 (595h26m51.16192072s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290862201 (595h26m51.161945257s) [seg_pos={0 0}, upload=521980]: Segment reached optimal threshold of 13 pieces. Dec 30 10:55:51.291085287 (595h26m51.162168352s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 10:55:51.291094224 (595h26m51.162177269s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291211416 (595h26m51.162294471s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:36672->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 10:55:51.291222287 (595h26m51.162305343s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291239801 (595h26m51.162322866s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection Dec 30 10:55:51.291248657 (595h26m51.162331702s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291276230 (595h26m51.162359275s) [seg_pos={0 0}, upload=521980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:55:51.291468056 (595h26m51.162551091s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:55:51.291477233 (595h26m51.162560278s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:55:51.589238178 (595h26m51.460321224s) [upload=521980]: Done uploading segments... Dec 30 10:55:51.589255381 (595h26m51.460338416s) [upload=521980]: Upload finished. err: <nil> Duration: 5.979034255s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX..........XX...X.X..XX.XX........X.......X..XX..X.X.X....] duration: 28.381274265s trace: https://jaeger.dev.storj.io/trace/4462c9db966cc2a FAIL (2025-12-30 10:59:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA) context deadline exceeded ======LOGS====== 7 (595h26m47.753396192s) [upload=521980]: Done starting upload Dec 30 10:55:47.882409701 (595h26m47.753492746s) [upload=521980]: Uploading segments... Dec 30 10:55:47.882622086 (595h26m47.753705141s) [upload=521980]: Got next segment. Inline: false Dec 30 10:55:47.882677732 (595h26m47.753760777s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:55:47.882684334 (595h26m47.753767379s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:55:48.132811478 (595h26m48.003894513s) [seg_pos={0 0}, upload=521980]: Begin upload segment... Dec 30 10:55:48.132886200 (595h26m48.003969245s) [seg_pos={0 0}, upload=521980]: Done begin upload segment. Dec 30 10:55:48.132935223 (595h26m48.004018258s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133006459 (595h26m48.004089494s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133505349 (595h26m48.004588394s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133867649 (595h26m48.004950694s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134246290 (595h26m48.005329335s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134718358 (595h26m48.005801404s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134724330 (595h26m48.005807365s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135049399 (595h26m48.006132434s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135164258 (595h26m48.006247293s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135396590 (595h26m48.006479626s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135505588 (595h26m48.006588633s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135738722 (595h26m48.006821767s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135845255 (595h26m48.0069283s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136209859 (595h26m48.007292904s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136633235 (595h26m48.00771627s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.137244088 (595h26m48.008327123s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:49.839444246 (595h26m49.710527281s) [upload=521980]: Next returned nil segment Dec 30 10:55:49.839461207 (595h26m49.710544273s) [upload=521980]: Waiting for error group managing segments... Dec 30 10:55:50.117467287 (595h26m49.988550332s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.117565533 (595h26m49.988648578s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.122601262 (595h26m49.993684307s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.124079047 (595h26m49.995162082s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.129932483 (595h26m50.001015528s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.133574788 (595h26m50.004657823s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.138294364 (595h26m50.009377399s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.145152304 (595h26m50.016235339s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.148344040 (595h26m50.019427086s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.209083830 (595h26m50.080166865s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.507112905 (595h26m50.37819595s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.692343985 (595h26m50.56342703s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290837664 (595h26m51.16192072s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290862201 (595h26m51.161945257s) [seg_pos={0 0}, upload=521980]: Segment reached optimal threshold of 13 pieces. Dec 30 10:55:51.291085287 (595h26m51.162168352s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 10:55:51.291094224 (595h26m51.162177269s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291211416 (595h26m51.162294471s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:36672->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 10:55:51.291222287 (595h26m51.162305343s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291239801 (595h26m51.162322866s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection Dec 30 10:55:51.291248657 (595h26m51.162331702s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291276230 (595h26m51.162359275s) [seg_pos={0 0}, upload=521980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:55:51.291468056 (595h26m51.162551091s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:55:51.291477233 (595h26m51.162560278s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:55:51.589238178 (595h26m51.460321224s) [upload=521980]: Done uploading segments... Dec 30 10:55:51.589255381 (595h26m51.460338416s) [upload=521980]: Upload finished. err: <nil> Duration: 28.381308148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X....X.........X....................X.................] duration: 44.642549804s trace: https://jaeger.dev.storj.io/trace/427bb74fbf2823be FAIL (2025-12-30 10:59:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.712977061 (595h16m21.584060096s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713064618 (595h16m21.584147663s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713493665 (595h16m21.5845767s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713746215 (595h16m21.58482925s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713961054 (595h16m21.585044099s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714248462 (595h16m21.585331507s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714290352 (595h16m21.585373387s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714513316 (595h16m21.585596361s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714907127 (595h16m21.585990172s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.715511768 (595h16m21.586594813s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.716418374 (595h16m21.587501419s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718018150 (595h16m21.589101185s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718481222 (595h16m21.589564257s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719042791 (595h16m21.590125836s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719436842 (595h16m21.590519887s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:23.159616720 (595h16m23.030699765s) [upload=521821]: Next returned nil segment Dec 30 10:45:23.159642560 (595h16m23.030725605s) [upload=521821]: Waiting for error group managing segments... Dec 30 10:45:23.568018017 (595h16m23.439101062s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.583184015 (595h16m23.45426706s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.586325036 (595h16m23.457408081s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.806152585 (595h16m23.67723563s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.819009537 (595h16m23.690092582s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.845285812 (595h16m23.716368857s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.847447017 (595h16m23.718530062s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.856608672 (595h16m23.727691707s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.971619162 (595h16m23.842702197s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.974330555 (595h16m23.84541363s) [noise=true , address=145.79.227.18:4009 , piece_id=74G4RMIC7XOUQII3FVQYAVNIF2AKTQWTK4IEGNEZE7UDBFGQ4WPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.027797295 (595h16m23.89888033s) [noise=true , address=45.91.116.4:4959 , piece_id=2NO7XKBMQASIHCJXHEJAURKY63F7TX3DCNXOPDIQLA5QUJW4MAXQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.429331223 (595h16m24.300414258s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.477331123 (595h16m24.348414178s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545180822 (595h16m24.416263857s) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545205568 (595h16m24.416288603s) [seg_pos={0 0}, upload=521821]: Segment reached optimal threshold of 14 pieces. Dec 30 10:45:24.545446567 (595h16m24.416529622s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545455103 (595h16m24.416538138s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545641889 (595h16m24.416724935s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545651768 (595h16m24.416734813s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545678299 (595h16m24.416761344s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545687706 (595h16m24.416770751s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545858341 (595h16m24.416941386s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection Dec 30 10:45:24.545866877 (595h16m24.416949912s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545980965 (595h16m24.41706401s) [seg_pos={0 0}, upload=521821]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:45:24.546075204 (595h16m24.417158239s) [upload=521821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:45:24.546084552 (595h16m24.417167587s) [upload=521821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:45:24.833013734 (595h16m24.704096779s) [upload=521821]: Done uploading segments... Dec 30 10:45:24.833037870 (595h16m24.704120915s) [upload=521821]: Upload finished. err: <nil> Duration: 44.642563912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [....X.X............X......X.X.XXXX.X.XX.X.X.....XX.X........] duration: 3.821959704s trace: https://jaeger.dev.storj.io/trace/18e339385ecbf6ab FAIL (2025-12-30 10:59:03 UTC) orbiter warning: context deadline exceeded Duration: 3.821984122s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.....X.X............X......X.X.XXXX.X.XX.X.X.....XX.X.......] duration: 3.471470426s trace: https://jaeger.dev.storj.io/trace/18e339385ecbf6ab FAIL (2025-12-30 10:58:27 UTC) orbiter warning: context deadline exceeded Duration: 3.471496145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XX..........XX...X.X..XX.XX........X.......X..XX..X.X.X...] duration: 13.202481717s trace: https://jaeger.dev.storj.io/trace/4462c9db966cc2a FAIL (2025-12-30 10:58:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA) context deadline exceeded ======LOGS====== 7 (595h26m47.753396192s) [upload=521980]: Done starting upload Dec 30 10:55:47.882409701 (595h26m47.753492746s) [upload=521980]: Uploading segments... Dec 30 10:55:47.882622086 (595h26m47.753705141s) [upload=521980]: Got next segment. Inline: false Dec 30 10:55:47.882677732 (595h26m47.753760777s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:55:47.882684334 (595h26m47.753767379s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:55:48.132811478 (595h26m48.003894513s) [seg_pos={0 0}, upload=521980]: Begin upload segment... Dec 30 10:55:48.132886200 (595h26m48.003969245s) [seg_pos={0 0}, upload=521980]: Done begin upload segment. Dec 30 10:55:48.132935223 (595h26m48.004018258s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133006459 (595h26m48.004089494s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133505349 (595h26m48.004588394s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.133867649 (595h26m48.004950694s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134246290 (595h26m48.005329335s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134718358 (595h26m48.005801404s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.134724330 (595h26m48.005807365s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135049399 (595h26m48.006132434s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135164258 (595h26m48.006247293s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135396590 (595h26m48.006479626s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135505588 (595h26m48.006588633s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135738722 (595h26m48.006821767s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.135845255 (595h26m48.0069283s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136209859 (595h26m48.007292904s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.136633235 (595h26m48.00771627s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:48.137244088 (595h26m48.008327123s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Uploading piece... Dec 30 10:55:49.839444246 (595h26m49.710527281s) [upload=521980]: Next returned nil segment Dec 30 10:55:49.839461207 (595h26m49.710544273s) [upload=521980]: Waiting for error group managing segments... Dec 30 10:55:50.117467287 (595h26m49.988550332s) [noise=true , address=145.79.227.23:4547 , piece_id=C77RNU3KFOP7OMCKUXRI7K3N4RPZNS2LGAD3FIZMCJY7QGMWHTMQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.117565533 (595h26m49.988648578s) [noise=true , address=145.79.227.13:4804 , piece_id=7HPHZXFSNAPDO4MJJ4F7LVXX6ETABICCUA6T6RZB4H6MN7YTITSA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.122601262 (595h26m49.993684307s) [noise=true , address=145.79.227.28:4726 , piece_id=V5Y4E56X5NWFWY3S4DDTG53KJ34YMFJEUPHL2TI6NYT34CEGLTPA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.124079047 (595h26m49.995162082s) [noise=true , address=145.79.227.14:4839 , piece_id=7DN2ZPKVYQGQ2AMVGAQK6VTDCJ2UYYHR4CNBU4J7NPA7CRUM3HAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.129932483 (595h26m50.001015528s) [noise=true , address=145.79.227.46:4602 , piece_id=XFO6KYMDZFIHKQU4N43LJDXPEGG3K42PHNOD7UOK6ALT3NDSGCIA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.133574788 (595h26m50.004657823s) [noise=true , address=145.79.227.4:4186 , piece_id=LD3KY2TADECOLQXZBBMGQSFLK2RGVF3RXOSHHC22HW6UQYGLPCPQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.138294364 (595h26m50.009377399s) [noise=true , address=145.79.227.7:4294 , piece_id=OZ5W3KHZM23UITKHGNAWROQM5WZGCUDPKG54Z7TA65CXWKFM7DLQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.145152304 (595h26m50.016235339s) [noise=true , address=145.79.227.17:4971 , piece_id=CAGLE2CWMANDX6KJKCW4FSDQGWQEH3OROWZYSIHZQUSNO3CAU4ZA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.148344040 (595h26m50.019427086s) [noise=true , address=145.79.227.12:4765 , piece_id=TUIH5N64Y7PFH4W5I4MGP5UBNR6FEG2LYZ5AUTPNNNGUEPY5IHEQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.209083830 (595h26m50.080166865s) [noise=true , address=145.79.227.45:4229 , piece_id=IHCNCCENVYG5NJVSW6COIXRH3QNVQMWAJWLFKUNKKZ7TFA4MAEGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.507112905 (595h26m50.37819595s) [noise=true , address=77.90.31.12:4590 , piece_id=GXNSFIO6CCRPCGXM7BT4MJYXOA37PCJKI7TKP6AHJG6ZEJ3EWGCQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:50.692343985 (595h26m50.56342703s) [noise=true , address=45.91.116.12:4227 , piece_id=FCINPIMLO2MDJDKXLL4DBNZX6B2KTRGXWHP2X665ZJTYNQ2IHNBQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290837664 (595h26m51.16192072s) [noise=true , address=45.91.116.14:4192 , piece_id=S5UUV7UKC3S54X4ZAMAYKK23HPAWI6HHOHIB224IZZNYQNMEPXGA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: <nil> Dec 30 10:55:51.290862201 (595h26m51.161945257s) [seg_pos={0 0}, upload=521980]: Segment reached optimal threshold of 13 pieces. Dec 30 10:55:51.291085287 (595h26m51.162168352s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 10:55:51.291094224 (595h26m51.162177269s) [noise=true , address=145.79.227.43:4159 , piece_id=IMPHFKT7EYLJNTY3Z67UK7EYKKTHWL7BNOBXYYDKZ36KUKSIB72A, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291211416 (595h26m51.162294471s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: read tcp 188.166.225.170:36672->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 10:55:51.291222287 (595h26m51.162305343s) [noise=true , address=77.90.31.10:4518 , piece_id=GNV3OG3UOERA5BWUNEXI6WWTQKI3XZLITWITH2FCAFI2TAAIKVAA, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291239801 (595h26m51.162322866s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43672->45.91.116.8:4047: use of closed network connection Dec 30 10:55:51.291248657 (595h26m51.162331702s) [noise=true , address=45.91.116.8:4047 , piece_id=FPCN4X2VTC23SNZ4KC7YMP45OXA7V4DCUC6EIH5UAV3X2IOB4ADQ, seg_pos={0 0}, upload=521980]: Long tail cancellation due to optimal threshold Dec 30 10:55:51.291276230 (595h26m51.162359275s) [seg_pos={0 0}, upload=521980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:55:51.291468056 (595h26m51.162551091s) [upload=521980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:55:51.291477233 (595h26m51.162560278s) [upload=521980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:55:51.589238178 (595h26m51.460321224s) [upload=521980]: Done uploading segments... Dec 30 10:55:51.589255381 (595h26m51.460338416s) [upload=521980]: Upload finished. err: <nil> Duration: 13.202513076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X....X.........X....................X................] duration: 32.728867966s trace: https://jaeger.dev.storj.io/trace/427bb74fbf2823be FAIL (2025-12-30 10:58:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.712977061 (595h16m21.584060096s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713064618 (595h16m21.584147663s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713493665 (595h16m21.5845767s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713746215 (595h16m21.58482925s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713961054 (595h16m21.585044099s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714248462 (595h16m21.585331507s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714290352 (595h16m21.585373387s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714513316 (595h16m21.585596361s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714907127 (595h16m21.585990172s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.715511768 (595h16m21.586594813s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.716418374 (595h16m21.587501419s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718018150 (595h16m21.589101185s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718481222 (595h16m21.589564257s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719042791 (595h16m21.590125836s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719436842 (595h16m21.590519887s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:23.159616720 (595h16m23.030699765s) [upload=521821]: Next returned nil segment Dec 30 10:45:23.159642560 (595h16m23.030725605s) [upload=521821]: Waiting for error group managing segments... Dec 30 10:45:23.568018017 (595h16m23.439101062s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.583184015 (595h16m23.45426706s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.586325036 (595h16m23.457408081s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.806152585 (595h16m23.67723563s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.819009537 (595h16m23.690092582s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.845285812 (595h16m23.716368857s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.847447017 (595h16m23.718530062s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.856608672 (595h16m23.727691707s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.971619162 (595h16m23.842702197s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.974330555 (595h16m23.84541363s) [noise=true , address=145.79.227.18:4009 , piece_id=74G4RMIC7XOUQII3FVQYAVNIF2AKTQWTK4IEGNEZE7UDBFGQ4WPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.027797295 (595h16m23.89888033s) [noise=true , address=45.91.116.4:4959 , piece_id=2NO7XKBMQASIHCJXHEJAURKY63F7TX3DCNXOPDIQLA5QUJW4MAXQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.429331223 (595h16m24.300414258s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.477331123 (595h16m24.348414178s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545180822 (595h16m24.416263857s) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545205568 (595h16m24.416288603s) [seg_pos={0 0}, upload=521821]: Segment reached optimal threshold of 14 pieces. Dec 30 10:45:24.545446567 (595h16m24.416529622s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545455103 (595h16m24.416538138s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545641889 (595h16m24.416724935s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545651768 (595h16m24.416734813s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545678299 (595h16m24.416761344s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545687706 (595h16m24.416770751s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545858341 (595h16m24.416941386s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection Dec 30 10:45:24.545866877 (595h16m24.416949912s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545980965 (595h16m24.41706401s) [seg_pos={0 0}, upload=521821]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:45:24.546075204 (595h16m24.417158239s) [upload=521821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:45:24.546084552 (595h16m24.417167587s) [upload=521821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:45:24.833013734 (595h16m24.704096779s) [upload=521821]: Done uploading segments... Dec 30 10:45:24.833037870 (595h16m24.704120915s) [upload=521821]: Upload finished. err: <nil> Duration: 32.728884538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X....X.........X....................X...............] duration: 19.147662265s trace: https://jaeger.dev.storj.io/trace/427bb74fbf2823be FAIL (2025-12-30 10:57:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.712977061 (595h16m21.584060096s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713064618 (595h16m21.584147663s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713493665 (595h16m21.5845767s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713746215 (595h16m21.58482925s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713961054 (595h16m21.585044099s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714248462 (595h16m21.585331507s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714290352 (595h16m21.585373387s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714513316 (595h16m21.585596361s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714907127 (595h16m21.585990172s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.715511768 (595h16m21.586594813s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.716418374 (595h16m21.587501419s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718018150 (595h16m21.589101185s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718481222 (595h16m21.589564257s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719042791 (595h16m21.590125836s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719436842 (595h16m21.590519887s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:23.159616720 (595h16m23.030699765s) [upload=521821]: Next returned nil segment Dec 30 10:45:23.159642560 (595h16m23.030725605s) [upload=521821]: Waiting for error group managing segments... Dec 30 10:45:23.568018017 (595h16m23.439101062s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.583184015 (595h16m23.45426706s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.586325036 (595h16m23.457408081s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.806152585 (595h16m23.67723563s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.819009537 (595h16m23.690092582s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.845285812 (595h16m23.716368857s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.847447017 (595h16m23.718530062s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.856608672 (595h16m23.727691707s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.971619162 (595h16m23.842702197s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.974330555 (595h16m23.84541363s) [noise=true , address=145.79.227.18:4009 , piece_id=74G4RMIC7XOUQII3FVQYAVNIF2AKTQWTK4IEGNEZE7UDBFGQ4WPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.027797295 (595h16m23.89888033s) [noise=true , address=45.91.116.4:4959 , piece_id=2NO7XKBMQASIHCJXHEJAURKY63F7TX3DCNXOPDIQLA5QUJW4MAXQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.429331223 (595h16m24.300414258s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.477331123 (595h16m24.348414178s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545180822 (595h16m24.416263857s) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545205568 (595h16m24.416288603s) [seg_pos={0 0}, upload=521821]: Segment reached optimal threshold of 14 pieces. Dec 30 10:45:24.545446567 (595h16m24.416529622s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545455103 (595h16m24.416538138s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545641889 (595h16m24.416724935s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545651768 (595h16m24.416734813s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545678299 (595h16m24.416761344s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545687706 (595h16m24.416770751s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545858341 (595h16m24.416941386s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection Dec 30 10:45:24.545866877 (595h16m24.416949912s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545980965 (595h16m24.41706401s) [seg_pos={0 0}, upload=521821]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:45:24.546075204 (595h16m24.417158239s) [upload=521821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:45:24.546084552 (595h16m24.417167587s) [upload=521821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:45:24.833013734 (595h16m24.704096779s) [upload=521821]: Done uploading segments... Dec 30 10:45:24.833037870 (595h16m24.704120915s) [upload=521821]: Upload finished. err: <nil> Duration: 19.147679528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X....X.........X....................X..............] duration: 37.356662349s trace: https://jaeger.dev.storj.io/trace/427bb74fbf2823be FAIL (2025-12-30 10:57:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.712977061 (595h16m21.584060096s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713064618 (595h16m21.584147663s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713493665 (595h16m21.5845767s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713746215 (595h16m21.58482925s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713961054 (595h16m21.585044099s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714248462 (595h16m21.585331507s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714290352 (595h16m21.585373387s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714513316 (595h16m21.585596361s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714907127 (595h16m21.585990172s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.715511768 (595h16m21.586594813s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.716418374 (595h16m21.587501419s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718018150 (595h16m21.589101185s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718481222 (595h16m21.589564257s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719042791 (595h16m21.590125836s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719436842 (595h16m21.590519887s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:23.159616720 (595h16m23.030699765s) [upload=521821]: Next returned nil segment Dec 30 10:45:23.159642560 (595h16m23.030725605s) [upload=521821]: Waiting for error group managing segments... Dec 30 10:45:23.568018017 (595h16m23.439101062s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.583184015 (595h16m23.45426706s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.586325036 (595h16m23.457408081s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.806152585 (595h16m23.67723563s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.819009537 (595h16m23.690092582s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.845285812 (595h16m23.716368857s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.847447017 (595h16m23.718530062s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.856608672 (595h16m23.727691707s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.971619162 (595h16m23.842702197s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.974330555 (595h16m23.84541363s) [noise=true , address=145.79.227.18:4009 , piece_id=74G4RMIC7XOUQII3FVQYAVNIF2AKTQWTK4IEGNEZE7UDBFGQ4WPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.027797295 (595h16m23.89888033s) [noise=true , address=45.91.116.4:4959 , piece_id=2NO7XKBMQASIHCJXHEJAURKY63F7TX3DCNXOPDIQLA5QUJW4MAXQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.429331223 (595h16m24.300414258s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.477331123 (595h16m24.348414178s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545180822 (595h16m24.416263857s) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545205568 (595h16m24.416288603s) [seg_pos={0 0}, upload=521821]: Segment reached optimal threshold of 14 pieces. Dec 30 10:45:24.545446567 (595h16m24.416529622s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545455103 (595h16m24.416538138s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545641889 (595h16m24.416724935s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545651768 (595h16m24.416734813s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545678299 (595h16m24.416761344s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545687706 (595h16m24.416770751s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545858341 (595h16m24.416941386s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection Dec 30 10:45:24.545866877 (595h16m24.416949912s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545980965 (595h16m24.41706401s) [seg_pos={0 0}, upload=521821]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:45:24.546075204 (595h16m24.417158239s) [upload=521821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:45:24.546084552 (595h16m24.417167587s) [upload=521821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:45:24.833013734 (595h16m24.704096779s) [upload=521821]: Done uploading segments... Dec 30 10:45:24.833037870 (595h16m24.704120915s) [upload=521821]: Upload finished. err: <nil> Duration: 37.356692726s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X.....X.X............X......X.X.XXXX.X.XX.X.X.....XX.X......] duration: 4.245472783s trace: https://jaeger.dev.storj.io/trace/18e339385ecbf6ab FAIL (2025-12-30 10:57:15 UTC) orbiter warning: context deadline exceeded Duration: 4.245499083s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)